Migrating data from Yandex Data Streams using Yandex Data Transfer

With Data Transfer, you can transfer data from a Data Streams data stream to an Yandex Object Storage bucket.

1. Prepare a Data Streams data stream.
2. Prepare and activate the transfer.
3. Test the transfer.

If you no longer need the resources you created, delete them.

Getting startedGetting started

Prepare the infrastructure:

Prepare a Data Streams data streamPrepare a Data Streams data stream

1. Create a Data Streams data stream.

2. Send test data to the data stream. Use data from the vehicle sensors in JSON format as a message:

{
    "device_id":"iv9a94th6rzt********",
    "datetime":"2020-06-05T17:27:00",
    "latitude":"55.70329032",
    "longitude":"37.65472196",
    "altitude":"427.5",
    "speed":"0",
    "battery_voltage":"23.5",
    "cabin_temperature":"17",
    "fuel_level":null
}

Prepare and activate the transferPrepare and activate the transfer

1. Create a source endpoint:

   • Database type: Yandex Data Streams.

   • Endpoint parameters:

     • Connection settings:

       • Database: Select the Managed Service for YDB database from the list.
       • Stream: Specify the name of the Data Streams data stream.
       • Service account: Select or create a service account with the yds.editor role.

     • Advanced settings:

       • Conversion rules: JSON

       • Data scheme: JSON specification:

         Create and upload the json_schema.json file in JSON format:

         json_schema.json

         [
             {
                 "name": "device_id",
                 "type": "string"
             },
             {
                 "name": "datetime",
                 "type": "datetime"
             },
             {
                 "name": "latitude",
                 "type": "double"
             },
             {
                 "name": "longitude",
                 "type": "double"
             },
             {
                 "name": "altitude",
                 "type": "double"
             },
             {
                 "name": "speed",
                 "type": "double"
             },
             {
                 "name": "battery_voltage",
                 "type": "any"
             },
             {
                 "name": "cabin_temperature",
                 "type": "double"
             },
             {
                 "name": "fuel_level",
                 "type": "any"
             }
         ]
         

2. Create a target endpoint:

   • Database type: Object Storage

   • Endpoint settings:

     • Connection settings:

       • Bucket: Enter the name of the bucket in Object Storage.
       • Service account: Select or create a service account with the storage.uploader role.

     • Serialization format: Select JSON or CSV if you have enabled Conversion rules in the advanced settings for the source endpoint.

3. Create a transfer:

   Manually

   Terraform

   1. Create a transfer of the Replication type that will use the created endpoints.
   2. Activate your transfer.

   1. In the data-transfer-yds-obj.tf file, uncomment:

      • Variables:

        • source_endpoint_id and set it to the value of the endpoint ID for the source.
        • target_endpoint_id and set it to the value of the endpoint ID for the target.

      • yandex_datatransfer_transfer resource.

   2. Make sure the Terraform configuration files are correct using this command:

      terraform validate
      

      If there are any errors in the configuration files, Terraform will point them out.

   3. Create the required infrastructure:

      1. Run the command to view planned changes:

         terraform plan
         

         If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.

      2. If you are happy with the planned changes, apply them:

         1. Run the command:

            terraform apply
            

         2. Confirm the update of resources.

         3. Wait for the operation to complete.

      Once created, your transfer will be activated automatically.

Test the transferTest the transfer

1. Wait for the transfer status to change to Replicating.

2. Make sure that the data from the Data Streams stream has been moved to the bucket Object Storage:

   1. In the management console, select the folder where the bucket is located.
   2. In the list of services, select Object Storage.
   3. Select the bucket from the list.
   4. Check that the bucket contains the <stream_name>_0.raw file (.json or .csv, depending on the selected output format) with the test data.

3. Send a new message to a Data Streams stream:

   {
       "device_id": "rhibbh3y08qm********",
       "datetime": "2020-06-06 09:49:54",
       "latitude": 55.71294467,
       "longitude": 37.66542005,
       "altitude": 429.13,
       "speed": 55.5,
       "battery_voltage": null,
       "cabin_temperature": 18,
       "fuel_level": 32
   }
   

4. Make sure that the following data has been added to the Object Storage bucket:

   1. In the management console, select the folder where the bucket is located.
   2. In the list of services, select Object Storage.
   3. Select the bucket from the list.
   4. Check that the <stream_name>_0.raw file (.json or .csv, depending on the selected output format) has been added to the bucket and includes the new data.

Delete the resources you createdDelete the resources you created

Some resources are not free of charge. To avoid paying for them, delete the resources you no longer need:

1. Delete the transfer.
2. Delete endpoints for both the source and target.
3. Delete the objects from the Object Storage bucket:

Delete the other resources depending on how they were created: