Migrating data to Yandex Object Storage using Yandex Data Transfer
With Data Transfer, you can transfer data from a Data Streams data stream to an Yandex Object Storage bucket.
If you no longer need the resources you created, delete them.
Getting started
Prepare the infrastructure:
-
Create a Managed Service for YDB database in any suitable configuration.
-
If you do not have Terraform yet, install it.
-
Get the authentication credentials. You can add them to environment variables or specify them later in the provider configuration file.
-
Configure and initialize a provider. There is no need to create a provider configuration file manually, you can download it
. -
Place the configuration file in a separate working directory and specify the parameter values. If you did not add the authentication credentials to environment variables, specify them in the configuration file.
-
Download the data-transfer-yds-obj.tf
configuration file to the same working directory.This file describes:
- Managed Service for YDB database.
- A service account with the
yds.editor
,storage.editor
, andstorage.uploader
roles. - Bucket in Object Storage.
- Transfer.
-
In the
data-transfer-yds-obj.tf
file, specify:folder_id
: ID of the folder to create resources in.sa_name
: Name of the service account to create a bucket in and use in endpoints.source_db_name
: Managed Service for YDB database name.bucket_name
: Object Storage bucket name.
-
Make sure the Terraform configuration files are correct using this command:
terraform validate
If there are any errors in the configuration files, Terraform will point them out.
-
Create the required infrastructure:
-
Run the command to view planned changes:
terraform plan
If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.
-
If you are happy with the planned changes, apply them:
-
Run the command:
terraform apply
-
Confirm the update of resources.
-
Wait for the operation to complete.
-
All the required resources will be created in the specified folder. You can check resource availability and their settings in the management console
. -
Prepare a Data Streams data stream
-
Send test data to the data stream. Use data from the vehicle sensors in JSON format as a message:
{
"device_id":"iv9a94th6rzt********",
"datetime":"2020-06-05T17:27:00",
"latitude":"55.70329032",
"longitude":"37.65472196",
"altitude":"427.5",
"speed":"0",
"battery_voltage":"23.5",
"cabin_temperature":"17",
"fuel_level":null
}
Prepare and activate the transfer
-
-
Database type:
Yandex Data Streams
. -
Endpoint parameters:
-
Connection settings:
- Database: Select the Managed Service for YDB database from the list.
- Stream: Specify the name of the Data Streams data stream.
- Service account: Select or create a service account with the
yds.editor
role.
-
Advanced settings:
-
Conversion rules:
JSON
-
Data scheme:
JSON specification
:Create and upload the
json_schema.json
file in JSON format:json_schema.json
[ { "name": "device_id", "type": "string" }, { "name": "datetime", "type": "datetime" }, { "name": "latitude", "type": "double" }, { "name": "longitude", "type": "double" }, { "name": "altitude", "type": "double" }, { "name": "speed", "type": "double" }, { "name": "battery_voltage", "type": "any" }, { "name": "cabin_temperature", "type": "double" }, { "name": "fuel_level", "type": "any" } ]
-
-
-
-
-
Database type:
Object Storage
-
Endpoint settings:
-
Connection settings:
- Bucket: Enter the name of the bucket in Object Storage.
- Service account: Select or create a service account with the
storage.uploader
role.
-
Serialization format: Select
JSON
orCSV
if you have enabled Conversion rules in the advanced settings for the source endpoint.
-
-
-
Create a transfer:
ManuallyTerraform- Create a transfer of the Replication type that will use the created endpoints.
- Activate your transfer.
-
In the
data-transfer-yds-obj.tf
file, uncomment:-
Variables:
source_endpoint_id
and set it to the value of the endpoint ID for the source.target_endpoint_id
and set it to the value of the endpoint ID for the target.
-
yandex_datatransfer_transfer
resource.
-
-
Make sure the Terraform configuration files are correct using this command:
terraform validate
If there are any errors in the configuration files, Terraform will point them out.
-
Create the required infrastructure:
-
Run the command to view planned changes:
terraform plan
If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.
-
If you are happy with the planned changes, apply them:
-
Run the command:
terraform apply
-
Confirm the update of resources.
-
Wait for the operation to complete.
-
Once created, your transfer will be activated automatically.
-
Test the transfer
-
Wait for the transfer status to change to Replicating.
-
Make sure that the data from the Data Streams stream has been moved to the bucket Object Storage:
- In the management console
, select the folder where the bucket is located. - In the list of services, select Object Storage.
- Select the bucket from the list.
- Check that the bucket contains the
<stream_name>_0.raw
file (.json
or.csv
, depending on the selected output format) with the test data.
- In the management console
-
Send a new message to a Data Streams stream:
{ "device_id": "rhibbh3y08qm********", "datetime": "2020-06-06 09:49:54", "latitude": 55.71294467, "longitude": 37.66542005, "altitude": 429.13, "speed": 55.5, "battery_voltage": null, "cabin_temperature": 18, "fuel_level": 32 }
-
Make sure that the following data has been added to the Object Storage bucket:
- In the management console
, select the folder where the bucket is located. - In the list of services, select Object Storage.
- Select the bucket from the list.
- Check that the
<stream_name>_0.raw
file (.json
or.csv
, depending on the selected output format) has been added to the bucket and includes the new data.
- In the management console
Delete the resources you created
Note
Before deleting the created resources, deactivate the transfer.
Some resources are not free of charge. To avoid paying for them, delete the resources you no longer need:
- Delete the transfer.
- Delete endpoints for both the source and target.
- Delete the objects from the Object Storage bucket:
Delete the other resources depending on how they were created:
- Delete the Managed Service for YDB database.
- Delete the bucket in Object Storage.
- If you created any service accounts when creating your endpoints, delete them.
-
In the terminal window, go to the directory containing the infrastructure plan.
-
Delete the
data-transfer-yds-obj.tf
configuration file. -
Make sure the Terraform configuration files are correct using this command:
terraform validate
If there are any errors in the configuration files, Terraform will point them out.
-
Confirm updating the resources.
-
Run the command to view planned changes:
terraform plan
If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.
-
If you are happy with the planned changes, apply them:
-
Run the command:
terraform apply
-
Confirm the update of resources.
-
Wait for the operation to complete.
-
All the resources defined in the
data-transfer-yds-obj.tf
configuration file will be deleted. -