Classification of images in video frames

Yandex DataSphere allows you to build ML models using the Jupyter Notebook interface.

This tutorial solves the problem of binary image classification. You may have to deal with it when recognizing vehicle types based on CCTV camera images. It is assumed that the CCTV system captures images from cameras when detecting motion and saves them to a bucket in Yandex Object Storage.

To get an idea of how the problem might be solved:

1. Prepare your infrastructure.
2. Configure DataSphere.
3. Create secrets.
4. Prepare notebooks.
5. Install the dependencies.
6. Upload and label the data.
7. Prepare the ML model and calculate the features.
8. Train the classifier using the features you calculated.
9. Predict the feature for the test image.
10. View practical applications of the model.

If you no longer need the resources you created, delete them.

Getting started

Before getting started, register in Yandex Cloud, set up a community, and link your billing account to it.

1. On the DataSphere home page, click Try for free and select an account to log in with: Yandex ID or your working account with the identity federation (SSO).
2. Select the Yandex Cloud Organization organization you are going to use in Yandex Cloud.
3. Create a community.
4. Link your billing account to the DataSphere community you are going to work in. Make sure you have a linked billing account and its status is ACTIVE or TRIAL_ACTIVE. If you do not have a billing account yet, create one in the DataSphere interface.

Required paid resources

The model operation cost includes:

• Fee for bucket usage (see Yandex Object Storage pricing).
• Fee for computing resource usage (see Yandex DataSphere pricing).

Prepare the infrastructure

Log in to the Yandex Cloud management console and select the organization you use to access DataSphere. On the Yandex Cloud Billing page, make sure you have a billing account linked.

If you have an active billing account, you can create or select a folder to deploy your infrastructure in, on the cloud page.

Create a folder to work in

Create a folder and network with subnets in each availability zone.

Create a service account

Create a static access key for the service account

To allow your service account to get authenticated in Object Storage, create a static access key.

Create an egress NAT gateway

Create an S3 bucket

Configure DataSphere

Create a project

1. Open the DataSphere home page.
2. In the left-hand panel, select Communities.
3. Select the community to create a project in.
4. On the community page, click Create project.
5. In the window that opens, enter a name and description (optional) for the project.
6. Click Create.

Edit the project settings

1. Select the relevant project in your community or on the DataSphere homepage in the Recent projects tab.

2. Navigate to the Settings tab.

3. Under Advanced settings, click Edit.

4. Specify the parameters:

   • Default folder: data-folder.
   • Service account: sa-for-project.
   • Subnet: Subnet of the ru-central1-a availability zone in data-folder.

   Note

   If you specified a subnet in the project settings, the time to allocate computing resources may be increased.

   • Security group: Specify a security group, if used in your organization.

5. Click Save.

Create secrets

Create secrets to store the ID and secret part of the static access key:

1. Under Project resources on the project page, click Secret.
2. Click Create.
3. In the Name field, enter a name for the secret. Name the secret with the static key ID as token, and the secret with the secret part as key_value.
4. In the Value field, enter a value to store in encrypted form.
5. Click Create. You will see a page with detailed info on the secret you created.

Prepare notebooks

Clone the Git repository containing the notebooks with the examples of the ML model training and testing:

1. In the top menu, click Git and select Clone.
2. In the window that opens, enter https://github.com/yandex-cloud-examples/yc-datasphere-video-recognition.git and click Clone.

Wait until cloning is complete. It may take some time. You will see the cloned repository folder in the File Browser section.

There are two notebooks in the repository:

• model-building.ipynb: To set up the environment and train your model using the ResNet50 convolutional neural network (CNN).
• model-testing.ipynb: To test your model.

Install dependencies

1. Open the ML folder and then the model-building.ipynb notebook.

2. Click on the first cell to select it:

   #!g1.1
   %matplotlib inline
   import matplotlib
   import matplotlib.pyplot as plt
   import os
   import io
   from os import path
   ...
   

3. To run the cell, click or press Shift + Enter.

4. Wait for the operation to complete.

The solution uses the Keras interface of the TensorFlow library with a CNTK backend. To connect to the bucket with images, you need the boto3 package. The cell also sets the environment variables needed to access the CNTK backend and connect to the bucket.

The packages listed in the cell are already installed in DataSphere and you can import them using the import command. For the full list of packages pre-installed in DataSphere, see List of pre-installed software.

Upload and label the data

Go to the Connect S3 section. You can use it to perform the following operations:

1. Set up a connection to the S3 bucket.
2. Load the list of objects, i.e., images of cars and buses. These will be used to train the model.
3. Define the function to extract an image using a key (name).

The next section, Labeling, is used for labeling data:

• Images are labeled according to the key value (folder name).
• Bus images are labeled with 0, and car images are labeled with 1.

To upload and label the data:

1. In the first cell, replace the bucket_name variable value with the name of your bucket. The default value is bucketwithvideo.

2. Select all the cells containing code in the Connect S3 and Labeling sections. To do this, hold Shift and click to the left of the cells in question:

   #!g1.1
   
   session = boto3.session.Session()
   ...
   

3. Run the selected cells.

4. Wait for the operation to complete. When the operation is complete, one of the images is displayed to check if the data was uploaded and labeled correctly.

Prepare the ML model and calculate the features

Go to the Calculating the characteristics section. You can use it to perform the following operations:

1. Load the ResNet50 model from the Keras package, with weights pretrained on the ImageNet dataset. This dataset contains 1.2 million images in 1,000 categories.
2. Define the featurize_images function. It splits the list of images into chunks of 32 images each, scales the images down to 224×224 pixels, and converts them to a four-dimensional tensor to feed to the Keras model. Next, the function calculates their features and returns them as a NumPy array.
3. Use the function to calculate binary features (1 means car, 0 means other) and save them to a file. This step may take 10-15 minutes. You can read more about the ResNet50 model here.

To prepare the model and calculate the features:

1. Select all the cells containing code in the Calculating the characteristics section:

   #!g1.1
   model = ResNet50(weights='imagenet',  input_shape=(224, 224, 3))
   ...
   

2. Run the selected cells.

3. Wait for the operation to complete.

Train the classifier using the features you calculated

In this section, you will train the LightGBM classifier on the features calculated in the previous section. To evaluate the model, you will use cross-validation.

The K-fold method of the scikit-learn package is used. Parts of the training set (folds) are created based on the specified percentage of data from each class.

This is important when your data contains much fewer images of one category than another. This example uses five-fold cross-validation. You can set a different number of folds in the n_splits parameter.

Go to the Training and cross-validation and Saving the model sections. These are used to perform the following operations:

1. Define an object for cross-validation of training results using the K-fold method.
2. Prepare a table to store the classification quality metrics.
3. Define the classification_metrics function for calculating the selected metrics.
4. Start the LightGBM classifier training. This example uses cross-validation comprising five folds:
   1. The training sample is divided into five equal non-overlapping parts.
   2. Five iterations take place. At each one, the following steps are performed:
      1. The model is trained on four parts from the sample.
      2. The model is tested on the sample part not used for training.
      3. The selected quality metrics are output.
5. Train the classifier on the full dataset and output the resulting error matrix.

To train the classifier, run all the cells in the Training and cross-validation and Saving the model sections one by one.

The model thus trained will be saved to a separate file.

Get the results of feature prediction based on test data

To use the resulting model:

1. Open the ML folder and then the model-testing.ipynb notebook.

   This notebook uses the previously trained LightGBM classifier to prepare the entities required to illustrate how you can use your model.

   Note

   You need much less resources to use your model than to train it; therefore, the minimum c1.4 configuration is left for this purpose (default).

2. In the second cell, replace the bucket_name variable value with the name of your bucket. The default value is bucketwithvideo.

3. Run the first three cells. Use these cells to:

   1. Import the packages you need for the test.
   2. Identify the name of the bucket to store the images.
   3. Set up a connection to the bucket with CCTV images.

4. Specify a test image featuring a car:

   test_image = 'car/electric-cars-17.jpeg'
   

   Note

   If you run the cell again, specify a new image.

5. In the cell from the Prediction section, load the ResNet50 model and the prepared LightGBM classifier and calculate the probability of the predicted binary feature value (1 means a car).

   It takes longer to process the cell with prediction calculation for the first time, because the models are loaded into the memory. The subsequent runs will be executed faster:

   %%time
   clf = lgb.Booster(model_file='ImageClassificationML/lightgbm_classifier.model')
   model = ResNet50(weights='imagenet',  input_shape=(224, 224, 3))
   ...
   

6. Make sure that the probability value is close to 1 (you should get ≈0.98).

7. Edit the cell code before loading the model:

   test_image = 'bus/electric_bus-183.jpeg'
   

   This is a test image with no car in it.

8. Run the cell.

9. Repeat the probability calculation and make sure that the value is much less than 0.5.

This means the classifier can successfully predict the feature for both images.

Practical use of the model

There are several practical uses for your model:

• The code of the proposed solution allows running a web service using Yandex Cloud Functions and analyze on-event CCTV content.
• For parallel processing of images collected from multiple video cameras into an S3 bucket, you can upload the code to an Apache Spark™ cluster in Yandex Data Processing using the PySpark package.

How to delete the resources you created

Some resources are not free of charge. To shut down the model and stop paying for the resources you created, delete those you no longer need:

1. Delete all objects from the bucket.
2. Delete the bucket.
3. Delete the project.
4. Delete the route table.
5. Delete the NAT gateway.