Transferring data to a Yandex Data Streams target endpoint

• Database: Select a Yandex Managed Service for YDB database registered in Yandex Data Streams as a target.

• Stream: Specify the name of the data stream associated with the database.

• Service account: Select or create a service account with the yds.editor role that Data Transfer will use to connect to the data target.

• Security groups: Select the cloud network to host the endpoint and security groups for network traffic.

  Thus, you will be able to apply the specified security group rules to the VMs and clusters in the selected network without changing the settings of these VMs and clusters. For more information, see Networking in Yandex Data Transfer.

• Save transaction order: Do not split an event stream into independent queues by table.

• Compresion codecs: Set the required data compression format (GZIP, ZSTD, or Raw).

• Under Serializing settings, select the serialization type:

  • Auto: Automatic serialization.

  • Debezium: Serialization under the Debezium standards:

    • Select the message key schema (matches the key.converter Debezium parameter).
    • Select the message value schema (matches the value.converter Debezium parameter).
    • If required, specify Debezium serializer settings in Parameter-Value format.

If you want to use JSON schemas in Yandex Schema Registry and preserve their compatibility when adding and deleting optional fields, use these settings:

• Serializing settings: Debezium.

• To use Schema Registry for keys, select Key schema type: JSON (via Schema Registry). To use Schema Registry for values, select Value schema type: JSON (via Schema Registry).

  • URL: Schema Registry namespace endpoint. You can copy the endpoint from the details for the Schema Registry namespace connection on the Debezium tab, in the value.converter.schema.registry.url parameter.

    Warning

    The namespace must have Compatibility check policy for JSON set to optional friendly.

  • Username: api-key.

  • Password: Value of the API key with a limited scope used for connecting to Schema Registry. To get this value:

    1. Create an API key with a limited scope and place it in the SECRET local variable:

       yc iam api-key create --folder-id <folder_ID> \
         --service-account-name <name_of_service_account_for_operations_with_Schema_Registry> \
         --scopes yc.schema-registry.schemas.manage \
         --expires-at '2030-01-01T00:00:00Z' >./api-key.yaml && \
       SECRET=`cat ./api-key.yaml | grep 'secret:' | awk '{print $2}'`
       

    2. Print the SECRET variable value to the terminal:

       echo $SECRET
       

    3. Copy the printed value and paste it into the Password field in the endpoint creation window.

• Under Debezium serializer settings:

  • To generate a closed schema for keys, include the key.converter.dt.json.generate.closed.content.schema parameter set to true.
  • To generate a closed schema for values, include the value.converter.dt.json.generate.closed.content.schema parameter set to true.