Creating a trigger for Data Streams that invokes a Serverless Containers container

If you do not have the Yandex Cloud CLI installed yet, install and initialize it.

By default, the CLI uses the folder specified when creating the profile. To change the default folder, use the yc config set folder-id <folder_ID> command. You can also set a different folder for any specific command using the --folder-name or --folder-id parameter.

To create a trigger that invokes a container, run this command:

yc serverless trigger create yds \
  --name <trigger_name> \
  --database <database_location> \
  --stream <stream_name> \
  --batch-size <message_batch_size> \
  --batch-cutoff <maximum_timeout> \
  --stream-service-account-id <service_account_ID> \
  --invoke-container-id <container_ID> \
  --invoke-container-service-account-id <service_account_ID> \
  --retry-attempts <number_of_retry_attempts> \
  --retry-interval <interval_between_retry_attempts> \
  --dlq-queue-id <dead-letter_queue_ID> \
  --dlq-service-account-id <service_account_ID>

Where:

• --name: Trigger name.

• --database: Location of the YDB DB the Data Streams stream is linked to.

  To find out where the DB is located, run the yc ydb database list command. The DB location is specified in the ENDPOINT column, in the database parameter, e.g., /ru-central1/b1gia87mbah2********/etn7hehf6gh3********.

• --stream: Stream name.

• --batch-size: Message batch size. This is an optional parameter. The values may range from 1 B to 64 KB. The default value is 1 B.

• --batch-cutoff: Maximum wait time. This is an optional parameter. The values may range from 1 to 60 seconds. The default value is 1 second. The trigger groups messages for a period not exceeding batch-cutoff and sends them to a container. The total amount of the data transmitted to a container may exceed batch-size if the data is transmitted as a single message. In all other cases, the amount of data does not exceed batch-size.

• --stream-service-account-id: ID of the service account with permissions to read from the stream and write to it.

• --invoke-container-id: Container ID.
• --invoke-container-service-account-id: ID of the service account with permissions to invoke the container.
• --retry-attempts: Number of invocation retries before the trigger moves a message to the dead-letter queue. This is an optional parameter. The values may range from 1 to 5. The default value is 1.
• --retry-interval: Time to retry invoking the container if the current attempt fails. This is an optional parameter. The values may range from 10 to 60 seconds. The default value is 10 seconds.
• --dlq-queue-id: Dead-letter queue ID. This is an optional parameter.
• --dlq-service-account-id: ID of the service account with write permissions to the dead-letter queue. This is an optional parameter.

Result:

id: a1s5msktijh2********
folder_id: b1gmit33hgh2********
created_at: "2022-10-24T14:07:04.693126923Z"
name: data-streams-trigger
rule:
  data_stream:
    database: /ru-central1/b1gia87mbah2********/etn7hehh2********
    stream: streams-name
    service_account_id: ajep8qm0kh2********
    batch_settings:
      size: "1"
      cutoff: 1s
    invoke_container:
      container_id: bba5jb38o8h2********
      service_account_id: aje03adgd2h2********
      retry_settings:
        retry_attempts: "1"
        interval: 10s
      dead_letter_queue:
        queue-id: yrn:yc:ymq:ru-central1:b1gmit33ngh2********:dlq
        service-account-id: aje3lebfemh2********
status: ACTIVE

With Terraform, you can quickly create a cloud infrastructure in Yandex Cloud and manage it using configuration files. These files store the infrastructure description written in HashiCorp Configuration Language (HCL). If you change the configuration files, Terraform automatically detects which part of your configuration is already deployed, and what should be added or removed.

Terraform is distributed under the Business Source License. The Yandex Cloud provider for Terraform is distributed under the MPL-2.0 license.

For more information about the provider resources, see the relevant documentation on the Terraform website or its mirror.

If you do not have Terraform yet, install it and configure the Yandex Cloud provider.

To create a trigger for Data Streams:

1. In the configuration file, describe the trigger parameters:

   resource "yandex_function_trigger" "my_trigger" {
     name = "<trigger_name>"
     container {
       id                 = "<container_ID>"
       service_account_id = "<service_account_ID>"
       retry_attempts     = "<number_of_retry_attempts>"
       retry_interval     = "<time_between_retry_attempts>"
     }
     data_streams {
       stream_name        = "<stream_name>"
       database           = "<database_location>"
       service_account_id = "<service_account_ID>"
       batch_cutoff       = "<maximum_timeout>"
       batch_size         = "<message_group_size>"
     }
     dlq {
       queue_id           = "<dead-letter_queue_ID>"
       service_account_id = "<service_account_ID>"
     }
   }
   

   Where:

   • name: Trigger name. The name format is as follows:

     • It must be from 2 to 63 characters long.
     • It can only contain lowercase Latin letters, numbers, and hyphens.
     • It must start with a letter and cannot end with a hyphen.

   • container: Container parameters:

     • id: Container ID.
     • service_account_id: ID of the service account with rights to invoke the container.
     • retry_attempts: Number of invocation retries before the trigger moves a message to the dead letter queue. This is an optional parameter. The values may range from 1 to 5. The default value is 1.
     • retry_intervall: Time to retry invoking the container if the current attempt fails. This is an optional parameter. The values may range from 10 to 60 seconds. The default value is 10 seconds.

   • data_streams: Trigger parameters:

     • stream_name: Stream name.

     • database: Location of the YDB DB the Data Streams stream is linked to.

       To find out where the DB is located, run the yc ydb database list command. The DB location is specified in the ENDPOINT column, in the database parameter, e.g., /ru-central1/b1gia87mba**********/etn7hehf6g*******.

     • service_account_id: ID of the service account with permissions to read from the stream and write to it.

     • batch_cutoff: Maximum wait time. This is an optional parameter. The values may range from 1 to 60 seconds. The default value is 1 second. The trigger groups messages for a period not exceeding batch_cutoff and sends them to a container. The number of messages cannot exceed batch_size.

     • batch_size: Message batch size. This is an optional parameter. The values may range from 1 B to 64 KB. The default value is 1 B.

   • dlq: Dead-letter queue message parameters:
     • queue_id: Dead-letter queue ID. This is an optional parameter.
     • service_account_id: ID of the service account with write permissions to the dead-letter queue. This is an optional parameter.

   For more information about the yandex_function_trigger resource parameters, see the relevant provider documentation.

2. Create resources:

   1. In the terminal, go to the directory where you edited the configuration file.

   2. Make sure the configuration file is correct using this command:

      terraform validate
      

      If the configuration is correct, you will get this message:

      Success! The configuration is valid.
      

   3. Run this command:

      terraform plan
      

      You will see a detailed list of resources. No changes will be made at this step. If the configuration contains any errors, Terraform will show them.

   4. Apply the changes:

      terraform apply
      

   5. Type yes and press Enter to confirm the changes.

   Terraform will create all the required resources. You can check the new resources using the management console or this CLI command:

   yc serverless trigger list
   

To create a trigger for Data Streams, use the create REST API method for the Trigger resource or the TriggerService/Create gRPC API call.