Creating a change data feed for a table

Change data capture (CDC) is a process of tracking changes in a particular table and delivering them to a consumer. When you add, update, or delete a table row, CDC generates a change record with the primary key of the row and writes it to the topic partition matching this key. A topic is an entity for storing unstructured messages and delivering them to multiple subscribers. Basically, a topic is a named set of messages.

Description of the yandex_ydb_table_changefeed resourceDescription of the yandex_ydb_table_changefeed resource

You can create a change data feed (CDF) for a table using the yandex_ydb_table_changefeed resource:

resource "yandex_ydb_table_changefeed" "ydb_changefeed" {
  table_id = yandex_ydb_table.test_table_2.id
  name = "changefeed"
  mode = "NEW_IMAGE"
  format = "JSON"

  consumer {
    name = "test_consumer"
  }
}

We used the following fields to create a table CDF:

• table_id: ID of the table to create a CDF for.
• name: CDF name.
• mode: CDF mode. For all available CDF modes, see this guide.
• format: CDF format. Only JSON format is available.

Full description of all the yandex_ydb_table_changefeed resource fields:

When initializing the yandex_ydb_table_changefeed resource, you can only specify one connection field: connection_string, table_path, or table_id. If you specify multiple connection fields, they will come into conflict, e.g., the table_id field with a relative link in <resource>.<ID>.<parameter> format: yandex_ydb_table.test_table_2.id.

The CDF resource contains the consumer section. A consumer is a named entity for reading data from the topic. Each consumer has several setting fields. The main field is name, i.e., the consumer's name. When initializing the yandex_ydb_table_changefeed resource, you can specify multiple consumers, or none; however, in this case, you will not be able to read data from the CDF.

Full list of the consumer section fields:

The consumer name is used in the SDK or CLI to read data from the topic.