Creating a trigger for Data Streams that invokes a container from Serverless Containers

Written by

Yandex Cloud

Improved by

Danila N.

Updated at September 30, 2026

View in Markdown

Getting started
Creating a trigger
Checking the result

Create a trigger for Data Streams that invokes a container from Serverless Containers when data is sent to a stream.

Note

The trigger for Data Streams receives and sends messages in JSON format only.

Getting started

To create a trigger, you will need:

Container the trigger will invoke. If you do not have a container:
- Create a container.
- Create a container revision.
Optionally, a dead-letter queue for unprocessed messages from the container. If you do not have a queue, create one.
Service accounts with the following permissions:
- To invoke a container.
- To read from the stream that will fire the trigger when data is sent to it.
- Optionally, to write to a dead-letter queue.
You can use the same service account or different ones. If you do not have a service account, create one.
Stream that will fire the trigger when data is sent to it. If you do not have a stream, create one.

Creating a trigger

Note

The trigger is initiated within five minutes after it is created.

Management console

CLI

Terraform

API

In the management console, select the folder where you want to create your trigger.
Navigate to Serverless Containers.
In the left-hand panel, select Triggers.
Click Create trigger.
Under Basic settings:
- Enter a name and description for the trigger.
- In the Labels field, click Add label and specify the labels in key: value format.
- In the Type field, select Data Streams.
Under Data Streams settings, select a data stream and a service account with read and write permissions for that stream.
Under Batch message settings, specify the following:
- Message batch size in bytes. The values may range from 1 B to 64 KB. The default value is 1 B.
- Maximum wait time. The values may range from 1 to 60 seconds. The default value is 1 second.
The trigger groups events within the specified wait time and sends them to the target. The total amount of data transmitted to connections may exceed the specified batch size if the data is transmitted as a single message. In all other cases, the amount of data does not exceed the batch size.
Under Targets:
1. In the Target type field, select Container.
2. Under Container settings, select a container and specify a service account that will invoke it.
3. Optionally, under Repeat request settings:
  - In the Interval field, specify the time to wait before retrying the container invocation if it fails. The values may range from 10 to 60 seconds. The default value is 10 seconds.
  - In the Number of attempts field, specify the number of invocation retries before the trigger moves a message to the dead-letter queue. The values may range from 1 to 5. The default value is 1.
4. Optionally, under Dead Letter Queue settings, select a dead-letter queue and a service account with write permissions for that queue.
5. Optionally, in the Filter field, specify a jq template to filter events sent to the target. If no filter is specified, all events are sent to the target.
6. Optionally, in the Transformation template field, specify a jq template to transform events before sending them to the target. If no template is specified, no transformations apply to the events.
Click Create trigger.

If you do not have the Yandex Cloud CLI yet, install and initialize it.

The folder used by default is the one specified when creating the CLI profile. To change the default folder, use the yc config set folder-id <folder_ID> command. You can also specify a different folder for any command using --folder-name or --folder-id.

If you access a resource by its name, the search will be limited to the default folder. If you access a resource by its ID, the search will be global, i.e., through all folders based on access permissions.

To create a trigger that invokes a container, run this command:

yc serverless trigger create yds \
  --name <trigger_name> \
  --database <database_location> \
  --stream <data_stream_name> \
  --batch-size <message_batch_size> \
  --batch-cutoff <maximum_wait_time> \
  --stream-service-account-id <service_account_ID> \
  --invoke-container-id <container_ID> \
  --invoke-container-service-account-id <service_account_ID> \
  --retry-attempts <number_of_retry_attempts> \
  --retry-interval <interval_between_retry_attempts> \
  --dlq-queue-id <dead-letter_queue_ID> \
  --dlq-service-account-id <service_account_ID>

Where:

--name: Trigger name.
--database: Location of the YDB database associated with the stream in Data Streams.

To find out where the database is located, run the yc ydb database list command. The database location is specified in the ENDPOINT column, in the database property, e.g., /ru-central1/b1gia87mbah2********/etn7hehf6gh3********.
--stream: Stream name.
--batch-size: Message batch size. This is an optional setting. The values may range from 1 B to 64 KB. The default value is 1 B.
--batch-cutoff: Maximum wait time. This is an optional setting. The values may range from 1 to 60 seconds. The default value is 1 second. The trigger groups messages within the batch-cutoff period and sends them to the container. The total amount of data transmitted to a container may exceed batch-size if the data is transmitted as a single message. In all other cases, the amount of data does not exceed batch-size.
--stream-service-account-id: ID of the service account with write and read permissions for the stream.

--invoke-container-id: Container ID.
--invoke-container-service-account-id: ID of the service account with permissions to invoke the container.
--retry-attempts: Number of invocation retries before the trigger moves a message to the dead-letter queue. This is an optional setting. The values may range from 1 to 5. The default value is 1.
--retry-interval: Time to wait before retrying the container invocation if it fails. This is an optional setting. The values may range from 10 to 60 seconds. The default value is 10 seconds.
--dlq-queue-id: Dead-letter queue ID. This is an optional setting.
--dlq-service-account-id: ID of the service account with write permissions for the dead-letter queue. This is an optional setting.

Result:

id: a1s5msktijh2********
folder_id: b1gmit33hgh2********
created_at: "2022-10-24T14:07:04.693126923Z"
name: data-streams-trigger
rule:
  data_stream:
    database: /ru-central1/b1gia87mbah2********/etn7hehh2********
    stream: streams-name
    service_account_id: ajep8qm0kh2********
    batch_settings:
      size: "1"
      cutoff: 1s
    invoke_container:
      container_id: bba5jb38o8h2********
      service_account_id: aje03adgd2h2********
      retry_settings:
        retry_attempts: "1"
        interval: 10s
      dead_letter_queue:
        queue-id: yrn:yc:ymq:ru-central1:b1gmit33ngh2********:dlq
        service-account-id: aje3lebfemh2********
status: ACTIVE

With Terraform, you can quickly create a cloud infrastructure in Yandex Cloud and manage it using configuration files. These files store the infrastructure description written in HashiCorp Configuration Language (HCL). If you change the configuration files, Terraform automatically detects which part of your configuration is already deployed, and what should be added or removed.

Terraform is distributed under the Business Source License. The Yandex Cloud provider for Terraform is distributed under the MPL-2.0 license.

For more information about the provider resources, see the guides on the Terraform website or its mirror.

If you do not have Terraform yet, install it and configure the Yandex Cloud provider.

To manage infrastructure using Terraform under a service account or user accounts (a Yandex account, a federated account, or a local user), authenticate using the appropriate method.

To create a trigger for Data Streams:

Describe the trigger in the configuration file:
```
resource "yandex_serverless_triggers" "my_trigger" {
  name = "<trigger_name>"
  source {
    yds {
      stream             = "<data_stream_name>"
      database           = "<database_location>"
      consumer           = "<consumer_name>"
      service_account_id = "<service_account_ID>"
      batch_settings {
        max_count = "<max_number_of_messages>"
        max_bytes = "<max_group_size_in_bytes>"
        cutoff    = "<maximum_wait_time>"
      }
    }
  }
  action {
    invoke_container {
      container_id       = "<container_ID>"
      path               = "<HTTP_path>"
      service_account_id = "<service_account_ID>"
    }
    retry_policy {
      retry_attempts = "<number_of_retries>"
      interval       = "<interval_between_retries>"
    }
    dead_letter {
      dead_letter_queue {
        queue_arn          = "<Dead_Letter_Queue_ARN>"
        service_account_id = "<service_account_ID>"
      }
    }
  }
}
```
Where:
- name: Trigger name. The name format is as follows:
  - Length: between 3 and 63 characters.
  - It can only contain lowercase Latin letters, numbers, and hyphens.
  - It must start with a letter and cannot end with a hyphen.
- description: Trigger description. This is an optional parameter.
- labels: Trigger labels in key:value format. This is an optional parameter.
- source: Event source settings:
  - yds: Data stream settings:
    - stream: Data stream name.
    - database: Location of the YDB database associated with the stream in Data Streams.
      
      To find out where the DB is located, run the yc ydb database list command. The database location is specified in the ENDPOINT column, in the database property, e.g., /ru-central1/b1gia87mba**********/etn7hehf6g*******.
    - consumer: Data stream consumer name.
    - service_account_id: ID of the service account with write and read permissions for the stream.
    - batch_settings: Event grouping settings. This is an optional section.
      - cutoff: Maximum event grouping time. This is a required setting. After the specified time has passed, the trigger sends the event group, even if it is not complete.
      - max_count: Maximum number of events per group.
      - max_bytes: Maximum total size of events per group, in bytes.
      At least one of the parameters, max_count or max_bytes, must be greater than 0.
- action: Target settings. You can specify this section multiple times so the trigger calls multiple resources, including those of different types. There are limits on the maximum number of resources.
  - invoke_container: Container settings:
    - container_id: Container ID.
    - path: HTTP path to call the container at. This is an optional parameter.
    - service_account_id: ID of the service account with permissions to invoke the container.
  - filter: Filtering events before sending them to the target. This is an optional section.
    - jq: jq template to filter events sent to the target. It not specified, all events reach the target.
  - transformer: Transforming events before sending them to the target. This is an optional section.
    - jq: jq template to transform events before sending them to the target. It omitted, no transformations apply to the events.
  - retry_policy: Repeated request settings. This is an optional section.
    - interval: Time interval before a retry attempt to send the event if the current attempt fails.
    - retry_attempts: Number of retry attempts before the trigger moves the event to the dead-letter queue.
  - dead_letter: Dead-letter queue settings. This is an optional section.
    - dead_letter_queue: Queue settings:
      - queue_arn: Queue ARN.
      - service_account_id: ID of the service account with permissions to write to the queue.
      - message_attributes: Attributes to add to each message in the queue, in key:value format. This is an optional parameter.
For more on the properties of the yandex_serverless_triggers resource, see this provider guide.
Configuration for the yandex_function_trigger resource
```
resource "yandex_function_trigger" "my_trigger" {
  name = "<trigger_name>"
  container {
    id                 = "<container_ID>"
    service_account_id = "<service_account_ID>"
    retry_attempts     = "<number_of_retry_attempts>"
    retry_interval     = "<time_between_retry_attempts>"
  }
  data_streams {
    stream_name        = "<data_stream_name>"
    database           = "<database_location>"
    service_account_id = "<service_account_ID>"
    batch_cutoff       = "<maximum_wait_time>"
    batch_size         = "<message_batch_size>"
  }
  dlq {
    queue_id           = "<dead-letter_queue_ID>"
    service_account_id = "<service_account_ID>"
  }
}
```
Where:
- name: Trigger name. Follow these naming requirements:
  - Length: between 3 and 63 characters.
  - It can only contain lowercase Latin letters, numbers, and hyphens.
  - It must start with a letter and cannot end with a hyphen.
- container: Container settings:
  - id: Container ID.
  - service_account_id: ID of the service account with permissions to invoke the container.
  - retry_attempts: Number of invocation retries before the trigger moves a message to the dead-letter queue. This is an optional setting. The values may range from 1 to 5. The default value is 1.
  - retry_interval: Time to wait before retrying the container invocation if it fails. This is an optional setting. The values may range from 10 to 60 seconds. The default value is 10 seconds.
- data_streams: Trigger settings:
  - stream_name: Data stream name.
  - database: Location of the YDB database associated with the stream in Data Streams.
    
    To find out where the database is located, run the yc ydb database list command. The database location is specified in the ENDPOINT column, in the database property, e.g., /ru-central1/b1gia87mba**********/etn7hehf6g*******.
  - service_account_id: ID of the service account with write and read permissions for the stream.
  - batch_cutoff: Maximum wait time. The values may range from 1 to 60 seconds. The default value is 1 second. The trigger groups messages within the batch_cutoff period and sends them to the container. The number of messages cannot exceed batch_size.
  - batch_size: Message batch size. This is an optional setting. The values may range from 1 B to 64 KB. The default value is 1 B.
- dlq: Dead-letter queue settings:
  - queue_id: Dead-letter queue ID.
  - service_account_id: ID of the service account with write permissions for the dead-letter queue.
For more on the properties of the yandex_function_trigger resource, see this provider guide.
Create the resources:
1. In the terminal, navigate to the configuration file directory.
2. Make sure the configuration is correct using this command:
```
terraform validate
```
  If the configuration is valid, you will get this message:
```
Success! The configuration is valid.
```
3. Run this command:
```
terraform plan
```
  You will see a list of resources and their properties. No changes will be made at this step. Terraform will show any errors in the configuration.
4. Apply the configuration changes:
```
terraform apply
```
5. Type yes and press Enter to confirm the changes.
Terraform will create all the required resources. You can check the new resources using the management console or this CLI command:
```
yc serverless trigger list
```

To create a trigger for Data Streams, use the create REST API method for the Trigger resource or the TriggerService/Create gRPC API call.

Checking the result

Check that the trigger works correctly. To do this, view container logs that show information on invocations.

Creating a trigger for Data Streams that invokes a container from Serverless Containers

Getting started

Creating a trigger

Checking the result

Useful links

Was the article helpful?

Creating a trigger for Data Streams that invokes a container from Serverless Containers

Getting startedGetting started

Creating a triggerCreating a trigger

Checking the resultChecking the result

Useful linksUseful links

Was the article helpful?

Getting started

Creating a trigger

Checking the result

Useful links