Use MLflow Experiments with Merlin containers[QST]

[hkristof03]

❓ Questions & Help

Hi everyone,

I am using the Tensorflow pre-built container to train Ranking models. I am trying to use MLflow to properly log the hyperparameters, results and artifacts but I am not able to install it. I saw it appearing for example here, but I haven't found an example and I could not install it in the container.

Details

Dockerfile:

FROM nvcr.io/nvidia/merlin/merlin-tensorflow:23.09

WORKDIR /workspace

RUN pip install mlflow

CMD ["jupyter-lab", "--allow-root", "--ip='0.0.0.0'", "--NotebookApp.token=''", "--no-browser"]

Error:

#0 9.122     Found existing installation: blinker 1.4
#0 9.123 ERROR: Cannot uninstall 'blinker'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.

I tried to install older MLflow versions but all attempts resulted in the error above. Is there a way to install MLflow in these pre-built containers and use it with Merlin? If it is already supported in a way, could you link the code / documentation / notebook?

[hkristof03]

@rnyak could you comment on this issue, please, or could you recommend a solution?

[EvenOldridge]

Hi @hkristof03 , unfortunately we don't have the bandwidth right now to support dependency interactions. Is the reason it's trying to uninstall because there's a conflict with the version of blinker? Are you able to update it manually to a newer version that's compatible with what MLflow requires?

[nv-alaiacano]

Try doing a (slightly unsafe) forced upgrade of blinker without first uninstalling it:

FROM nvcr.io/nvidia/merlin/merlin-tensorflow:23.09

WORKDIR /workspace

RUN pip install --ignore-installed blinker
RUN pip install mlflow

CMD ["jupyter-lab", "--allow-root", "--ip='0.0.0.0'", "--NotebookApp.token=''", "--no-browser"]

I was able to build this container and launch jupyterlab.

Inside jupyterlab I opened a terminal and started mlflow

$ mlflow server

Then create an experiment/run and log params successfully:

import mlflow

mlflow.set_tracking_uri("http://localhost:5000")

mlflow.create_experiment("my experiment")

with mlflow.start_run() as run:
    mlflow.log_param("log_paramsey", "value")

[rnyak]

@hkristof03 I am closing this issue. Please see the solution idea from @nv-alaiacano above. if you need to reopen this ticket, pls do so.

[pklemenkov]

This solution doesn't work as soon as you start using Tensorflow, because installation of that kind breaks pandas.
File "/usr/local/lib/python3.10/dist-packages/merlin/models/tf/__init__.py", line 20, in <module> from merlin.dataloader.tf_utils import configure_tensorflow File "/usr/local/lib/python3.10/dist-packages/merlin/dataloader/tf_utils.py", line 24, in <module> from merlin.core.dispatch import HAS_GPU File "/usr/local/lib/python3.10/dist-packages/merlin/core/dispatch.py", line 21, in <module> import dask.dataframe as dd File "/usr/local/lib/python3.10/dist-packages/dask/dataframe/__init__.py", line 4, in <module> from dask.dataframe import backends, dispatch, rolling File "/usr/local/lib/python3.10/dist-packages/dask/dataframe/backends.py", line 22, in <module> from dask.dataframe.core import DataFrame, Index, Scalar, Series, _Frame File "/usr/local/lib/python3.10/dist-packages/dask/dataframe/core.py", line 35, in <module> from dask.dataframe import methods File "/usr/local/lib/python3.10/dist-packages/dask/dataframe/methods.py", line 22, in <module> from dask.dataframe.utils import is_dataframe_like, is_index_like, is_series_like File "/usr/local/lib/python3.10/dist-packages/dask/dataframe/utils.py", line 19, in <module> from dask.dataframe import ( # noqa: F401 register pandas extension types File "/usr/local/lib/python3.10/dist-packages/dask/dataframe/_dtypes.py", line 4, in <module> from dask.dataframe.extensions import make_array_nonempty, make_scalar File "/usr/local/lib/python3.10/dist-packages/dask/dataframe/extensions.py", line 6, in <module> from dask.dataframe.accessor import ( File "/usr/local/lib/python3.10/dist-packages/dask/dataframe/accessor.py", line 190, in <module> class StringAccessor(Accessor): File "/usr/local/lib/python3.10/dist-packages/dask/dataframe/accessor.py", line 276, in StringAccessor pd.core.strings.StringMethods, AttributeError: module 'pandas.core.strings' has no attribute 'StringMethods'