Creating a trigger for Message Queue that sends messages to Cloud Functions

If you do not have the Yandex Cloud CLI yet, install and initialize it.

The folder used by default is the one specified when creating the CLI profile. To change the default folder, use the yc config set folder-id <folder_ID> command. You can also specify a different folder for any command using --folder-name or --folder-id.

If you access a resource by its name, the search will be limited to the default folder. If you access a resource by its ID, the search will be global, i.e., through all folders based on access permissions.

To create a trigger that invokes a function, run this command:

yc serverless trigger create message-queue \
  --name <trigger_name> \
  --queue <queue_ID> \
  --queue-service-account-id <service_account_ID> \
  --invoke-function-id <function_ID> \
  --invoke-function-service-account-id <service_account_ID> \
  --batch-size <message_batch_size> \
  --batch-cutoff <maximum_wait_time>

Where:

• --name: Trigger name.

• --queue: Queue ID.

  To find out the queue ID:

  1. In the management console, navigate to the folder containing the queue.
  2. Navigate to Message Queue.
  3. Select the queue.
  4. You can see the queue ID under General information in the ARN field.

• --invoke-function-id: Function ID.

• --queue-service-account-name: ID of the service account with permissions to read messages from the queue.

• --invoke-function-service-account-id: ID of the service account with permissions to invoke the function.

• --batch-size: Message batch size. This is an optional setting. The values may range from 1 to 1,000. The default value is 1.

• --batch-cutoff: Maximum wait time. This is an optional setting. The values may range from 0 to 20 seconds. The default value is 10 seconds. The trigger groups messages within the batch-cutoff period and sends them to the function. The number of messages cannot exceed batch-size.

Result:

id: dd0cspdch6**********
folder_id: aoek49ghmk**********
created_at: "2019-08-28T12:14:45.762915Z"
name: ymq-trigger
rule:
  message_queue:
    queue_id: yrn:yc:ymq:ru-central1:aoek49ghmk**********:my-mq
    service_account_id: bfbqqeo6jk**********
    batch_settings:
      size: "1"
      cutoff: 10s
    invoke_function:
      function_id: b09e5lu91t**********
      function_tag: $latest
      service_account_id: bfbqqeo6j**********
status: ACTIVE

With Terraform, you can quickly create a cloud infrastructure in Yandex Cloud and manage it using configuration files. These files store the infrastructure description written in HashiCorp Configuration Language (HCL). If you change the configuration files, Terraform automatically detects which part of your configuration is already deployed, and what should be added or removed.

Terraform is distributed under the Business Source License. The Yandex Cloud provider for Terraform is distributed under the MPL-2.0 license.

For more information about the provider resources, see the guides on the Terraform website or its mirror.

If you do not have Terraform yet, install it and configure the Yandex Cloud provider.

To manage infrastructure using Terraform under a service account or user accounts (a Yandex account, a federated account, or a local user), authenticate using the appropriate method.

To create a trigger for Message Queue:

1. Describe the trigger in the configuration file:

   resource "yandex_serverless_triggers" "my_trigger" {
     name        = "<trigger_name>"
     description = "<trigger_description>"
     source {
       ymq {
         queue_arn          = "<queue_ARN>"
         service_account_id = "<service_account_ID>"
         visibility_timeout = "<message_visibility_timeout>"
         batch_settings {
           max_count = "<max_number_of_messages>"
           max_bytes = "<max_group_size_in_bytes>"
           cutoff    = "<maximum_wait_time>"
         }
       }
     }
     action {
       invoke_function {
         function_id        = "<function_ID>"
         service_account_id = "<service_account_ID>"
       }
     }
   }
   

   Where:

   • name: Trigger name. The name format is as follows:

     • Length: between 3 and 63 characters.
     • It can only contain lowercase Latin letters, numbers, and hyphens.
     • It must start with a letter and cannot end with a hyphen.

   • description: Trigger description. This is an optional parameter.

   • labels: Trigger labels in key:value format. This is an optional parameter.

   • source: Event source settings:

     • ymq: Message queue settings:

       • queue_arn: Message queue ARN.

         To find out the queue ARN:

         1. In the management console, navigate to the folder containing the queue.
         2. Navigate to Message Queue.
         3. Select the queue.
         4. You can see the queue ARN under General information in the ARN field.

       • service_account_id: ID of the service account with permissions to read messages from the queue.

       • visibility_timeout: Message visibility timeout that overrides the value specified in the queue. This is an optional parameter.

       • batch_settings: Event grouping settings. This is an optional section.

         • cutoff: Maximum event grouping time. This is a required setting. After the specified time has passed, the trigger sends the event group, even if it is not complete.
         • max_count: Maximum number of events per group.
         • max_bytes: Maximum total size of events per group, in bytes.

         At least one of the parameters, max_count or max_bytes, must be greater than 0.

   • action: Target settings. You can specify this section multiple times so that the trigger calls multiple resources, including those of different types. There are limits on the maximum number of resources.

     • invoke_function: Function settings:

       • function_id: Function ID.
       • function_tag: Function version tag. This is an optional parameter. If it is not specified, the latest version of the function is called.
       • service_account_id: ID of the service account with permissions to invoke the function.

     • filter: Filtering events before sending them to the target. This is an optional section.

       • jq: jq template to filter events sent to the target. It not specified, all events reach the target.

     • transformer: Transforming events before sending them to the target. This is an optional section.

       • jq: jq template to transform events before sending them to the target. It omitted, no transformations apply to the events.

     • dead_letter: Dead-letter queue settings. This is an optional section.

       • dead_letter_queue: Queue settings:

         • queue_arn: Queue ARN.
         • service_account_id: ID of the service account with permissions to write to the queue.
         • message_attributes: Attributes to add to each message in the queue, in key:value format. This is an optional parameter.

   For more on the properties of the yandex_serverless_triggers resource, see this provider guide.

   Configuration for the yandex_function_trigger resource

   resource "yandex_function_trigger" "my_trigger" {
     name        = "<trigger_name>"
     description = "<trigger_description>"
     function {
       id                 = "<function_ID>"
       service_account_id = "<service_account_ID>"
     }
     message_queue {
       queue_id           = "<queue_ID>"
       service_account_id = "<service_account_ID>"
       batch_size         = "<message_batch_size>"
       batch_cutoff       = "<maximum_wait_time>"
   }
   

   Where:

   • name: Trigger name. The name format is as follows:

     • Length: between 3 and 63 characters.
     • It can only contain lowercase Latin letters, numbers, and hyphens.
     • It must start with a letter and cannot end with a hyphen.

   • description: Trigger description.

   • function: Function settings:

     • id: Function ID.
     • service_account_id: ID of the service account with permissions to invoke the function.

   • message_queue: Trigger settings:

     • queue_id: Message queue ID.

       To find out the queue ID:

       1. In the management console, navigate to the folder containing the queue.
       2. Navigate to Message Queue.
       3. Select the queue.
       4. You can see the queue ID under General information in the ARN field.

     • service_account_id: ID of the service account with permissions to read messages from the queue.

     • batch_size: Message batch size. This is an optional setting. The values may range from 1 to 1,000. The default value is 1.

     • batch_cutoff: Maximum wait time. This is an optional setting. The values may range from 0 to 20 seconds. The default value is 10 seconds. The trigger groups messages within the batch-cutoff period and sends them to the function. The number of messages cannot exceed batch-size.

   For more on the properties of the yandex_function_trigger resource, see this provider guide.

2. Create the resources:

   1. In the terminal, navigate to the configuration file directory.

   2. Make sure the configuration is correct using this command:

      terraform validate
      

      If the configuration is valid, you will get this message:

      Success! The configuration is valid.
      

   3. Run this command:

      terraform plan
      

      You will see a list of resources and their properties. No changes will be made at this step. Terraform will show any errors in the configuration.

   4. Apply the configuration changes:

      terraform apply
      

   5. Type yes and press Enter to confirm the changes.

   Terraform will create all the required resources. You can check the new resources using the management console or this CLI command:

   yc serverless trigger list
   

To create a trigger for Yandex Message Queue, use the create REST API method for the Trigger resource or the TriggerService/Create gRPC API call.

Check that the number of enqueued messages is decreasing. To do this, view the queue statistics:

1. Navigate to Message Queue.
2. Select the queue for which you created the trigger.
3. Go to Monitoring. Check the Messages in the queue chart.