Creating an MLFlow server for logging experiments and artifacts

This tutorial describes how to deploy an MLFlow tracking server for logging experiments and artifacts on a separate Yandex Compute Cloud VM. We will run experiments in JupyterLab Notebook. We will use a Yandex Managed Service for PostgreSQL database to store internal objects and a Yandex Object Storage bucket to store artifacts.

To create an MLFlow server for logging JupyterLab Notebook experiments and artifacts:

1. Set up your infrastructure.
2. Create a static access key.
3. Create an SSH key pair.
4. Create a VM.
5. Create a managed DB.
6. Create a bucket.
7. Install the MLFlow tracking server and add it to the VM auto start.
8. Create secrets.
9. Train your model.

If you no longer need the resources you created, delete them.

Getting startedGetting started

Before getting started, register in Yandex Cloud, set up a community, and link your billing account to it.

1. On the DataSphere home page, click Try for free and select an account to log in with: Yandex ID or your working account with the identity federation (SSO).
2. Select the Yandex Identity Hub organization you are going to use in Yandex Cloud.
3. Create a community.
4. Link your billing account to the DataSphere community you are going to work in. Make sure you have a linked billing account and its status is ACTIVE or TRIAL_ACTIVE. If you do not have a billing account yet, create one in the DataSphere interface.

Required paid resourcesRequired paid resources

The cost of training a model based on Object Storage data includes:

• Fee for using DataSphere computing resources.
• Fee for using Compute Cloud computing resources.
• Fee for a running Managed Service for PostgreSQL cluster.
• Fee for storing data in a bucket (see Object Storage pricing).
• Fee for data operations (see Object Storage pricing).

Set up your infrastructureSet up your infrastructure

Log in to the Yandex Cloud management console and select the organization you use to access DataSphere. On the Yandex Cloud Billing page, make sure you have a billing account linked.

If you have an active billing account, you can go to the cloud page to create or select a folder to run your infrastructure.

Create a folderCreate a folder

Create a service account for Object StorageCreate a service account for Object Storage

To access a bucket in Object Storage, you will need a service account with the storage.viewer and storage.uploader roles.

Create a static access keyCreate a static access key

To access Object Storage from DataSphere, you will need a static key.

Create an SSH key pairCreate an SSH key pair

To connect to a VM over SSH, you will need a key pair: the public key resides on the VM, and the private one is kept by the user. This method is more secure than login and password authentication.

To create a key pair:

Create a VMCreate a VM

Create a managed DBCreate a managed DB

Create a bucketCreate a bucket

Install the MLFlow tracking server and add it to the VM auto startInstall the MLFlow tracking server and add it to the VM auto start

1. Connect to the VM over SSH.

2. Download the Anaconda distribution:

   curl --remote-name https://repo.anaconda.com/archive/Anaconda3-2023.07-1-Linux-x86_64.sh
   

3. Run its installation:

   bash Anaconda3-2023.07-1-Linux-x86_64.sh
   

   Wait for the installation to complete and restart the shell.

4. Create an environment:

   conda create -n mlflow
   

5. Activate the environment:

   conda activate mlflow
   

6. Install the required packages by running these commands one by one:

   conda install -c conda-forge mlflow
   conda install -c anaconda boto3
   pip install psycopg2-binary
   pip install pandas
   

7. Create the environment variables for S3 access:

   • Open the file with the variables:

     sudo nano /etc/environment
     

   • Add these lines to the file while substituting the placeholders with your VM's internal IP address:

     MLFLOW_S3_ENDPOINT_URL=https://storage.yandexcloud.net/
     MLFLOW_TRACKING_URI=http://<VM_internal_IP_address>:8000
     

8. Specify the data for the boto3 library to access S3:

   • Create the .aws folder:

     mkdir ~/.aws
     

   • Create the credentials file:

     nano ~/.aws/credentials
     

   • Add these lines to the file while substituting the placeholders with the static key ID and value:

     [default]
     aws_access_key_id=<static_key_ID>
     aws_secret_access_key=<secret_key>
     

9. Run the MLFlow tracking server while substituting the placehoders with your cluster data:

   mlflow server --backend-store-uri postgresql://<username>:<password>@<host>:6432/db1?sslmode=verify-full --default-artifact-root s3://mlflow-bucket/artifacts -h 0.0.0.0 -p 8000
   

   You can check your connection to MLFlow at http://<VM_public_IP_address>:8000.

Enable MLFlow autorunEnable MLFlow autorun

For MLFlow to run automatically after the VM restarts, you need to convert it into a Systemd service.

1. Create directories for storing logs and error details:

   mkdir ~/mlflow_logs/
   mkdir ~/mlflow_errors/
   

2. Create the mlflow-tracking.service file:

   sudo nano /etc/systemd/system/mlflow-tracking.service
   

3. Add these lines to the file while substituting the placeholders with your data:

   [Unit]
   Description=MLflow Tracking Server
   After=network.target
   
   [Service]
   Environment=MLFLOW_S3_ENDPOINT_URL=https://storage.yandexcloud.net/
   Restart=on-failure
   RestartSec=30
   StandardOutput=file:/home/<VM_user_name>/mlflow_logs/stdout.log
   StandardError=file:/home/<VM_username>/mlflow_errors/stderr.log
   User=<VM_username>
   ExecStart=/bin/bash -c 'PATH=/home/<VM_username>/anaconda3/envs/mlflow_env/bin/:$PATH exec mlflow server --backend-store-uri postgresql://<DB_user_name>:<password>@<host>:6432/db1?sslmode=verify-full --default-artifact-root s3://mlflow-bucket/artifacts -h 0.0.0.0 -p 8000'
   
   [Install]
   WantedBy=multi-user.target
   

   Where:

   • <VM_user_name>: VM account user name.
   • <DB_user_name>: User name specified when creating the database cluster.

4. Run the service and enable autoload at system startup:

   sudo systemctl daemon-reload
   sudo systemctl enable mlflow-tracking
   sudo systemctl start mlflow-tracking
   sudo systemctl status mlflow-tracking
   

Create secretsCreate secrets

1. Select the project in your community or on the DataSphere home page in the Recent projects tab.

2. Under Project resources, click Secret.
3. Click Create.
4. In the Name field, enter the name for the secret: MLFLOW_S3_ENDPOINT_URL.
5. In the Value field, paste the URL: https://storage.yandexcloud.net/.
6. Click Create.
7. Create three more secrets:
   • MLFLOW_TRACKING_URI with the http://<VM_internal_IP_address>:8000 value.
   • AWS_ACCESS_KEY_ID with the static key ID.
   • AWS_SECRET_ACCESS_KEY with the static key value.

Train your modelTrain your model

This tutorial features a set of data for predicting the quality of wine based on quantitative properties, such as acidity, pH, residual sugar, etc. To train your model, copy and paste the code into the notebook cells.

1. Open the DataSphere project:

   1. Select the project in your community or on the DataSphere home page in the Recent projects tab.

   2. Click Open project in JupyterLab and wait for the loading to complete.
   3. Open the notebook tab.

2. Install the required modules:

   %pip install mlflow
   

3. Import the required libraries:

   import os
   import warnings
   import sys
   
   import pandas as pd
   import numpy as np
   from sklearn.metrics import mean_squared_error, mean_absolute_error, r2_score
   from sklearn.model_selection import train_test_split
   from sklearn.linear_model import ElasticNet
   from urllib.parse import urlparse
   import mlflow
   import mlflow.sklearn
   from mlflow.models import infer_signature
   import logging
   

4. Create an experiment in MLFlow:

   mlflow.set_experiment("my_first_experiment")
   

5. Create a function for prediction quality assessment:

   def eval_metrics(actual, pred):
     rmse = np.sqrt(mean_squared_error(actual, pred))
     mae = mean_absolute_error(actual, pred)
     r2 = r2_score(actual, pred)
     return rmse, mae, r2
   

6. Get your data ready, train your model, and register it in MLflow:

   logging.basicConfig(level=logging.WARN)
   logger = logging.getLogger(__name__)
   
   warnings.filterwarnings("ignore")
   np.random.seed(40)
   
   # Uploading the dataset to assess wine quality
   csv_url = (
      "https://raw.githubusercontent.com/mlflow/mlflow/master/tests/datasets/winequality-red.csv"
   )
   try:
       data = pd.read_csv(csv_url, sep=";")
   except Exception as e:
       logger.exception(
           "Unable to download training & test CSV, check your internet connection. Error: %s", e
      )
   
   # Splitting the dataset into a training sample and a test sample
   train, test = train_test_split(data)
   
   # Allocating a target variable and variables used for prediction
   train_x = train.drop(["quality"], axis=1)
   test_x = test.drop(["quality"], axis=1)
   train_y = train[["quality"]]
   test_y = test[["quality"]]
   
   alpha = 0.5
   l1_ratio = 0.5
   
   # Creating an `mlflow` run
   with mlflow.start_run():
     
      # Creating and training the ElasticNet model
      lr = ElasticNet(alpha=alpha, l1_ratio=l1_ratio, random_state=42)
      lr.fit(train_x, train_y)
     
      # Making quality predictions against the test sample
      predicted_qualities = lr.predict(test_x)
   
      (rmse, mae, r2) = eval_metrics(test_y, predicted_qualities)
   
      print("Elasticnet model (alpha={:f}, l1_ratio={:f}):".format(alpha, l1_ratio))
      print("  RMSE: %s" % rmse)
      print("  MAE: %s" % mae)
      print("  R2: %s" % r2)
   
      # Logging data on hyperparameters and quality metrics in MLflow
      mlflow.log_param("alpha", alpha)
      mlflow.log_param("l1_ratio", l1_ratio)
      mlflow.log_metric("rmse", rmse)
      mlflow.log_metric("r2", r2)
      mlflow.log_metric("mae", mae)
   
      predictions = lr.predict(train_x)
      signature = infer_signature(train_x, predictions)
   
      tracking_url_type_store = urlparse(mlflow.get_tracking_uri()).scheme
   
      # Registering the model in MLflow
      if tracking_url_type_store != "file":
        mlflow.sklearn.log_model(
              lr, "model", registered_model_name="ElasticnetWineModel", signature=signature
        )
      else:
         mlflow.sklearn.log_model(lr, "model", signature=signature)
   

   You can check the result at http://<VM_public_IP_address>:8000.

How to delete the resources you createdHow to delete the resources you created

To stop paying for the resources you created:

• Delete the VM.
• Delete the database cluster.
• Delete the objects from the bucket.
• Delete the bucket.
• Delete the project.