Creating an MLFlow server for logging experiments and artifacts

This tutorial describes how to deploy an MLFlow tracking server for logging experiments and artifacts on a separate Yandex Compute Cloud VM. We will run experiments in JupyterLab Notebook. We will use a Yandex Managed Service for PostgreSQL database to store internal objects and a Yandex Object Storage bucket to store artifacts.

To create an MLFlow server for logging JupyterLab Notebook experiments and artifacts:

1. Prepare your infrastructure.
2. Create a static access key.
3. Create an SSH key pair.
4. Create a VM.
5. Create a managed DB.
6. Create a bucket.
7. Install the MLFlow tracking server and add it to the VM auto-start.
8. Create secrets.
9. Train the model.

If you no longer need the resources you created, delete them.

Getting startedGetting started

Before getting started, register in Yandex Cloud, set up a community, and link your billing account to it.

1. On the DataSphere home page, click Try for free and select an account to log in with: Yandex ID or your working account in the identity federation (SSO).
2. Select the Yandex Cloud Organization organization you are going to use in Yandex Cloud.
3. Create a community.
4. Link your billing account to the DataSphere community you are going to work in. Make sure that you have a billing account linked and its status is ACTIVE or TRIAL_ACTIVE. If you do not have a billing account yet, create one in the DataSphere interface.

Required paid resourcesRequired paid resources

The cost of training a model based on data from Object Storage includes:

• Fee for DataSphere computing resource usage.
• Fee for Compute Cloud computing resource usage.
• Fee for a running Managed Service for PostgreSQL cluster.
• Fee for storing data in a bucket (see Object Storage pricing).
• Fee for operations with data (see Object Storage pricing).

Prepare the infrastructurePrepare the infrastructure

Log in to the Yandex Cloud management console and select the organization you use to access DataSphere. On the Yandex Cloud Billing page, make sure you have a billing account linked.

If you have an active billing account, you can create or select a folder to deploy your infrastructure in, on the cloud page.

Create a folderCreate a folder

Create a service account for Object StorageCreate a service account for Object Storage

To access a bucket in Object Storage, you will need a service account with the storage.viewer and storage.uploader roles.

Create a static access keyCreate a static access key

To access Object Storage from DataSphere, you need a static key.

Create an SSH key pairCreate an SSH key pair

To connect to a VM over SSH, you need a key pair: the public key resides on the VM, and the private one is kept by the user. This method is more secure than connecting with login and password.

To create a key pair:

Create a VMCreate a VM

Create a managed DBCreate a managed DB

Create a bucketCreate a bucket

Install the MLFlow tracking server and add it to the VM auto-startInstall the MLFlow tracking server and add it to the VM auto-start

1. Connect to the VM via SSH.

2. Download the Anaconda distribution:

   curl --remote-name https://repo.anaconda.com/archive/Anaconda3-2023.07-1-Linux-x86_64.sh
   

3. Run its installation:

   bash Anaconda3-2023.07-1-Linux-x86_64.sh
   

   Wait for the installation to complete and restart the shell.

4. Create an environment:

   conda create -n mlflow
   

5. Activate the environment:

   conda activate mlflow
   

6. Install the required packages by running the following commands one by one:

   conda install -c conda-forge mlflow
   conda install -c anaconda boto3
   pip install psycopg2-binary
   pip install pandas
   

7. Create environment variables for S3 access:

   • Open the file with the variables:

     sudo nano /etc/environment
     

   • Add the following lines to the file by substituting your VM's internal IP address:

     MLFLOW_S3_ENDPOINT_URL=https://storage.yandexcloud.net/
     MLFLOW_TRACKING_URI=http://<VM_internal_IP_address>:8000
     

8. Specify the data to be used by the boto3 library for S3 access:

   • Create the .aws folder:

     mkdir ~/.aws
     

   • Create the credentials file:

     nano ~/.aws/credentials
     

   • Add the following lines to the file by substituting the static key ID and value:

     [default]
     aws_access_key_id=<static_key_ID>
     aws_secret_access_key=<secret_key>
     

9. Run the MLFlow tracking server through this command (provide your cluster data instead of the palceholders):

   mlflow server --backend-store-uri postgresql://<username>:<password>@<host>:6432/db1?sslmode=verify-full --default-artifact-root s3://mlflow-bucket/artifacts -h 0.0.0.0 -p 8000
   

   You can check your connection to MLFlow at http://<VM_public_IP_address>:8000.

Enable MLFlow autorunEnable MLFlow autorun

For MLFlow to run automatically after the VM restarts, you need to convert it into a Systemd service.

1. Create directories for storing logs and error details:

   mkdir ~/mlflow_logs/
   mkdir ~/mlflow_errors/
   

2. Create the mlflow-tracking.service file:

   sudo nano /etc/systemd/system/mlflow-tracking.service
   

3. Add the following lines to the file, replacing the placeholders with your data:

   [Unit]
   Description=MLflow Tracking Server
   After=network.target
   
   [Service]
   Environment=MLFLOW_S3_ENDPOINT_URL=https://storage.yandexcloud.net/
   Restart=on-failure
   RestartSec=30
   StandardOutput=file:/home/<VM_user_name>/mlflow_logs/stdout.log
   StandardError=file:/home/<VM_user_name>/mlflow_errors/stderr.log
   User=<VM_user_name>
   ExecStart=/bin/bash -c 'PATH=/home/<VM_user_name>/anaconda3/envs/mlflow_env/bin/:$PATH exec mlflow server --backend-store-uri postgresql://<DB_user_name>:<password>@<host>:6432/db1?sslmode=verify-full --default-artifact-root s3://mlflow-bucket/artifacts -h 0.0.0.0 -p 8000'
   
   [Install]
   WantedBy=multi-user.target
   

   Where:

   • <VM_user_name>: VM account username.
   • <DB_user_name>: Username specified when creating a database cluster.

4. Run the service and enable autoload at system startup:

   sudo systemctl daemon-reload
   sudo systemctl enable mlflow-tracking
   sudo systemctl start mlflow-tracking
   sudo systemctl status mlflow-tracking
   

Create secretsCreate secrets

1. Select the relevant project in your community or on the DataSphere homepage in the Recent projects tab.

2. Under Project resources, click Secret.
3. Click Create.
4. In the Name field, enter the name for the secret: MLFLOW_S3_ENDPOINT_URL.
5. In the Value field, paste the URL: https://storage.yandexcloud.net/.
6. Click Create.
7. Create three more secrets:
   • MLFLOW_TRACKING_URI with the http://<VM_internal_IP_address>:8000 value.
   • AWS_ACCESS_KEY_ID with the static key ID.
   • AWS_SECRET_ACCESS_KEY with the static key value.

Train the modelTrain the model

The example uses a set of data to predict the quality of wine based on quantitative characteristics, such as acidity, PH, residual sugar, etc. To train the model, copy and paste the code into notebook cells.

1. Open the DataSphere project:

   1. Select the relevant project in your community or on the DataSphere homepage in the Recent projects tab.

   2. Click Open project in JupyterLab and wait for the loading to complete.
   3. Open the notebook tab.

2. Install the required modules:

   %pip install mlflow
   

3. Import the required libraries:

   import os
   import warnings
   import sys
   
   import pandas as pd
   import numpy as np
   from sklearn.metrics import mean_squared_error, mean_absolute_error, r2_score
   from sklearn.model_selection import train_test_split
   from sklearn.linear_model import ElasticNet
   from urllib.parse import urlparse
   import mlflow
   import mlflow.sklearn
   from mlflow.models import infer_signature
   import logging
   

4. Create an experiment in MLFlow:

   mlflow.set_experiment("my_first_experiment")
   

5. Create a function for prediction quality assessment:

   def eval_metrics(actual, pred):
     rmse = np.sqrt(mean_squared_error(actual, pred))
     mae = mean_absolute_error(actual, pred)
     r2 = r2_score(actual, pred)
     return rmse, mae, r2
   

6. Prepare data, train the model, and register it with MLflow:

   logging.basicConfig(level=logging.WARN)
   logger = logging.getLogger(__name__)
   
   warnings.filterwarnings("ignore")
   np.random.seed(40)
   
   # Uploading dataset to assess wine quality
   csv_url = (
      "https://raw.githubusercontent.com/mlflow/mlflow/master/tests/datasets/winequality-red.csv"
   )
   try:
       data = pd.read_csv(csv_url, sep=";")
   except Exception as e:
       logger.exception(
           "Unable to download training & test CSV, check your internet connection. Error: %s", e
      )
   
   # Splitting dataset into training and test samples
   train, test = train_test_split(data)
   
   # Allocating target variable and variables used for prediction
   train_x = train.drop(["quality"], axis=1)
   test_x = test.drop(["quality"], axis=1)
   train_y = train[["quality"]]
   test_y = test[["quality"]]
   
   alpha = 0.5
   l1_ratio = 0.5
   
   # Creating mlflow run
   with mlflow.start_run():
     
      # Creating and training the ElasticNet model
      lr = ElasticNet(alpha=alpha, l1_ratio=l1_ratio, random_state=42)
      lr.fit(train_x, train_y)
     
      # Making quality predictions against the test sample
      predicted_qualities = lr.predict(test_x)
   
      (rmse, mae, r2) = eval_metrics(test_y, predicted_qualities)
   
      print("Elasticnet model (alpha={:f}, l1_ratio={:f}):".format(alpha, l1_ratio))
      print("  RMSE: %s" % rmse)
      print("  MAE: %s" % mae)
      print("  R2: %s" % r2)
   
      # Logging data on hyperparameters and quality metrics in MLflow
      mlflow.log_param("alpha", alpha)
      mlflow.log_param("l1_ratio", l1_ratio)
      mlflow.log_metric("rmse", rmse)
      mlflow.log_metric("r2", r2)
      mlflow.log_metric("mae", mae)
   
      predictions = lr.predict(train_x)
      signature = infer_signature(train_x, predictions)
   
      tracking_url_type_store = urlparse(mlflow.get_tracking_uri()).scheme
   
      # Registering model in MLflow
      if tracking_url_type_store != "file":
        mlflow.sklearn.log_model(
              lr, "model", registered_model_name="ElasticnetWineModel", signature=signature
        )
      else:
         mlflow.sklearn.log_model(lr, "model", signature=signature)
   

   You can check the result at http://<VM_public_IP_address>:8000.

How to delete the resources you createdHow to delete the resources you created

To stop paying for the resources you created:

• Delete the VM.
• Delete the database cluster.
• Delete the objects from the bucket.
• Delete the bucket.
• Delete the project.