Classification of images in video frames
- Getting started
- Prepare the infrastructure
- Configure DataSphere
- Create secrets
- Prepare notebooks
- Install dependencies
- Upload and label the data
- Prepare the ML model and calculate the features
- Train the classifier using the features you calculated
- Get the results of feature prediction based on test data
- Practical use of the model
- How to delete the resources you created
Yandex DataSphere allows you to build ML models using the Jupyter Notebook interface.
This tutorial solves the problem of binary image classification. You may have to deal with it when recognizing vehicle types based on CCTV camera images. It is assumed that the CCTV system captures images from cameras when detecting motion and saves them to a bucket in Yandex Object Storage.
To get an idea of how the problem might be solved:
- Prepare your infrastructure.
- Configure DataSphere.
- Create secrets.
- Prepare notebooks.
- Install the dependencies.
- Upload and label the data.
- Prepare the ML model and calculate the features.
- Train the classifier using the features you calculated.
- Predict the feature for the test image.
- View practical applications of the model.
If you no longer need the resources you created, delete them.
Getting started
Before getting started, register in Yandex Cloud, set up a community, and link your billing account to it.
- On the DataSphere home page
, click Try for free and select an account to log in with: Yandex ID or your working account in the identity federation (SSO). - Select the Yandex Cloud Organization organization you are going to use in Yandex Cloud.
- Create a community.
- Link your billing account to the DataSphere community you are going to work in. Make sure that you have a billing account linked and its status is
ACTIVE
orTRIAL_ACTIVE
. If you do not have a billing account yet, create one in the DataSphere interface.
Required paid resources
The model operation cost includes:
- Fee for bucket usage (see Yandex Object Storage pricing).
- Fee for computing resource usage (see Yandex DataSphere pricing).
Prepare the infrastructure
Log in to the Yandex Cloud management console
If you have an active billing account, you can create or select a folder to deploy your infrastructure in, on the cloud page
Note
If you use an identity federation to access Yandex Cloud, billing details might be unavailable to you. In this case, contact your Yandex Cloud organization administrator.
Create a folder to work in
Create a folder and network with subnets in each availability zone.
- In the management console
, select a cloud and click Create folder. - Name your folder, e.g.,
data-folder
. - Click Create.
Create a service account
-
Go to
data-folder
: -
In the list of services, select Identity and Access Management.
-
Click Create service account.
-
Enter a name for the service account, e.g.,
sa-for-project
. -
Click Add role and assign the following roles to the service account:
storage.viewer
: To read data from the Object Storage bucket.vpc.gateways.user
: To use a subnet.
-
Click Create.
Create a static access key for the service account
To allow your service account to get authenticated in Object Storage, create a static access key.
- Go to
data-folder
. - In the list of services, select Identity and Access Management.
- In the left-hand panel, select
Service accounts. - In the list that opens, select the
sa-for-project
service account. - Click Create new key in the top panel.
- Select Create static access key.
- Enter a description of the key so that you can easily find it in the management console.
- Save the ID and secret key. The secret key is not saved in Yandex Cloud, so you will not be able to view it in the management console.
Create an egress NAT gateway
- In
data-folder
, select Virtual Private Cloud. - In the left-hand panel, select Gateways.
- Click Create and set the gateway parameters:
- Name the gateway, e.g.,
nat-for-cluster
. - Gateway Type: Egress NAT.
- Click Save.
- Name the gateway, e.g.,
Create a route table:
- In the left-hand panel, select Routing tables.
- Click Create and specify the route table parameters:
- Enter a name, e.g.,
route-table
. - Select the
data-folder
network. - Click Add a route.
- In the window that opens, select Gateway in the Next hop field.
- In the Gateway field, select the NAT gateway you created. The destination prefix will be propagated automatically.
- Click Add.
- Enter a name, e.g.,
- Click Create a routing table.
Link the route table to a subnet to route traffic from it via the NAT gateway:
- In the left-hand panel, select
Subnets. - In the row with the subnet you need, click
. - In the menu that opens, select Link routing table.
- In the window that opens, select the created table from the list.
- Click Link.
Create an S3 bucket
- Go to
data-folder
. - In the list of services, select Object Storage.
- Click Create bucket.
- On the bucket creation page:
-
Enter a name for the bucket according to the naming requirements.
Warning
Do not use buckets with periods in their names for connection.
-
In the Object read access, Object listing access, and Read access to settings fields, select Restricted.
-
Limit the maximum bucket size, if required.
-
- Click Create bucket to complete the operation.
Configure DataSphere
Create a project
- Open the DataSphere home page
. - In the left-hand panel, select
Communities. - Select the community to create a project in.
- On the community page, click
Create project. - In the window that opens, enter a name and description (optional) for the project.
- Click Create.
Edit the project settings
-
Select the relevant project in your community or on the DataSphere homepage
in the Recent projects tab. -
Go to the Settings tab.
-
Under Advanced settings, click
Edit. -
Specify the parameters:
- Default folder:
data-folder
. - Service account:
sa-for-project
. - Subnet: Subnet of the
ru-central1-a
availability zone indata-folder
.
Note
If you specified a subnet in the project settings, the time to allocate computing resources may be increased.
- Security group: Specify a security group, if used in your organization.
- Default folder:
-
Click Save.
Create secrets
Create secrets to store the ID and secret part of the static access key:
- Under Project resources on the project page, click
Secret. - Click Create.
- In the Name field, enter a name for the secret. Name the secret with the static key ID as
token
, and the secret with the secret part askey_value
. - In the Value field, enter a value to be stored in encrypted form.
- Click Create. This will display the created secret's info page.
Prepare notebooks
Clone the Git repository containing the notebooks with the examples of the ML model training and testing:
- In the top menu, click Git and select Clone.
- In the window that opens, enter
https://github.com/yandex-cloud-examples/yc-datasphere-video-recognition.git
and click Clone.
Wait until cloning is complete. It may take some time. You will see the cloned repository folder in the
There are two notebooks in the repository:
model-building.ipynb
: To set up the environment and train your model using the ResNet50 convolutional neural network (CNN).model-testing.ipynb
: To test your model.
Install dependencies
Note
In this example, the model is trained and tested using the g1.1 computing resource configuration. You can use a different configuration with a GPU. To do this, change the configuration in the code in all notebook cells as needed.
-
Open the ML folder and then the model-building.ipynb notebook.
-
Click on the first cell to select it:
#!g1.1 %matplotlib inline import matplotlib import matplotlib.pyplot as plt import os import io from os import path ...
-
To run the cell, click
or press Shift + Enter. -
Wait for the operation to complete.
The solution uses the Keras interfaceboto3
package. The cell also sets the environment variables needed to access the CNTK backend and connect to the bucket.
The packages listed in the cell are already installed in DataSphere and you can import them using the import
command. For the full list of packages pre-installed in DataSphere, see List of pre-installed software.
Upload and label the data
Go to the Connect S3 section. You can use it to perform the following operations:
- Set up a connection to the S3 bucket.
- Load the list of objects, i.e., images of cars and buses. These will be used to train the model.
- Define the function to extract an image using a key (name).
The next section, Labeling, is used for labeling data:
- Images are labeled according to the key value (folder name).
- Bus images are labeled with
0
, and car images are labeled with1
.
To upload and label the data:
-
In the first cell, replace the
bucket_name
variable value with the name of your bucket. The default value isbucketwithvideo
. -
Select all the cells containing code in the Connect S3 and Labeling sections. To do this, hold Shift and click to the left of the cells in question:
#!g1.1 session = boto3.session.Session() ...
-
Run the selected cells.
-
Wait for the operation to complete. When the operation is complete, one of the images is displayed to check if the data was uploaded and labeled correctly.
Prepare the ML model and calculate the features
Go to the Calculating the characteristics section. You can use it to perform the following operations:
- Load the ResNet50 model from the Keras package, with weights pretrained on the ImageNet dataset. This dataset contains 1.2 million images in 1,000 categories.
- Define the
featurize_images
function. It splits the list of images into chunks of 32 images each, scales the images down to 224×224 pixels, and converts them to a four-dimensional tensor to feed to the Keras model. Next, the function calculates their features and returns them as a NumPy array. - Use the function to calculate binary features (
1
means car,0
means other) and save them to a file. This step may take 10-15 minutes. You can read more about the ResNet50 model here .
To prepare the model and calculate the features:
-
Select all the cells containing code in the Calculating the characteristics section:
#!g1.1 model = ResNet50(weights='imagenet', input_shape=(224, 224, 3)) ...
-
Run the selected cells.
-
Wait for the operation to complete.
Train the classifier using the features you calculated
In this section, you will train the LightGBM classifier on the features calculated in the previous section. To evaluate the model, you will use cross-validation.
The K-fold
This is important when your data contains much fewer images of one category than another. This example uses five-fold cross-validation. You can set a different number of folds in the n_splits
parameter.
Go to the Training and cross-validation and Saving the model sections. These are used to perform the following operations:
- Define an object for cross-validation of training results using the K-fold method.
- Prepare a table to store the classification quality metrics.
- Define the
classification_metrics
function for calculating the selected metrics. - Start the LightGBM classifier training. This example uses cross-validation comprising five folds:
- The training sample is divided into five equal non-overlapping parts.
- Five iterations take place. At each one, the following steps are performed:
- The model is trained on four parts from the sample.
- The model is tested on the sample part not used for training.
- The selected quality metrics are output.
- Train the classifier on the full dataset and output the resulting error matrix.
To train the classifier, run all the cells in the Training and cross-validation and Saving the model sections one by one.
The model thus trained will be saved to a separate file.
Get the results of feature prediction based on test data
To use the resulting model:
-
Open the ML folder and then the model-testing.ipynb notebook.
This notebook uses the previously trained LightGBM classifier to prepare the entities required to illustrate how you can use your model.
Note
You need much less resources to use your model than to train it; therefore, the minimum c1.4 configuration is left for this purpose (default).
-
In the second cell, replace the
bucket_name
variable value with the name of your bucket. The default value isbucketwithvideo
. -
Run the first three cells. Use these cells to:
- Import the packages you need for the test.
- Identify the name of the bucket to store the images.
- Set up a connection to the bucket with CCTV images.
-
Specify a test image featuring a car:
test_image = 'car/electric-cars-17.jpeg'
Note
If you run the cell again, specify a new image.
-
In the cell from the Prediction section, load the ResNet50 model and the prepared LightGBM classifier and calculate the probability of the predicted binary feature value (
1
means a car).It takes longer to process the cell with prediction calculation for the first time, because the models are loaded into the memory. The subsequent runs will be executed faster:
%%time clf = lgb.Booster(model_file='ImageClassificationML/lightgbm_classifier.model') model = ResNet50(weights='imagenet', input_shape=(224, 224, 3)) ...
-
Make sure that the probability value is close to 1 (you should get
≈0.98
). -
Edit the cell code before loading the model:
test_image = 'bus/electric_bus-183.jpeg'
This is a test image with no car in it.
-
Run the cell.
-
Repeat the probability calculation and make sure that the value is much less than
0.5
.
This means the classifier can successfully predict the feature for both images.
Note
You can share your finished notebook of computations or export the whole project.
Practical use of the model
There are several practical uses for your model:
- The code of the proposed solution allows running a web service using Yandex Cloud Functions and analyze on-event CCTV content.
- For parallel processing of images collected from multiple video cameras into an S3 bucket, you can upload the code to an Apache Spark™ cluster in Yandex Data Processing using the PySpark package.
How to delete the resources you created
Some resources are not free of charge. To shut down the model and stop paying for the resources you created, delete those you no longer need: