Transferring data to an Apache Kafka® target endpoint

• Endpoint type: kafka_target.

• security_groups: Security groups for network traffic.

  Security group rules apply to a transfer. They allow opening up network access from the transfer VM to the cluster. For more information, see Networking in Yandex Data Transfer.

  Security groups must belong to the same network as the cluster.

  Note

  In Terraform, it is not required to specify a network for security groups.

• connection.cluster_id: ID of the cluster to connect to.

• auth: Authentication method used to connect to broker hosts:

  • sasl: SASL authentication.
  • no_auth: Without authentication.

Here is an example of the configuration file structure:

resource "yandex_datatransfer_endpoint" "<endpoint_name_in_Terraform>" {
  name = "<endpoint_name>"
  settings {
    kafka_target {
      security_groups = ["<list_of_security_group_IDs>"]
      connection {
        cluster_id = "<cluster_ID>"
      }
      auth {
        <authentication_method>
      }
      <topic_settings>
      <serialization_settings>
    }
  }
}

For more information, see the Terraform provider documentation.

• securityGroups: Security groups for network traffic, whose rules will apply to VMs and clusters without changing their settings. For more information, see Networking in Yandex Data Transfer.

• connection.clusterId: ID of the cluster to connect to.

• auth: Authentication method used to connect to broker hosts:

  • sasl: SASL authentication. The following parameters are required:

    • user: Name of the account Data Transfer will use to connect to the topic.
    • password.raw: Password for the account in text form.
    • mechanism: Hashing mechanism.

  • noAuth: Without authentication.

• Endpoint type: kafka_target.

• security_groups: Security groups for network traffic.

  Security group rules apply to a transfer. They allow opening up network access from the transfer VM to the broker hosts. For more information, see Networking in Yandex Data Transfer.

  Security groups must belong to the same network as the subnet_id subnet, if the latter is specified.

  Note

  In Terraform, it is not required to specify a network for security groups.

• connection.on_premise: Parameters for connecting to the broker hosts:

  • broker_urls: IP addresses or FQDNs of the broker hosts.

    If the Apache Kafka® port number differs from the standard one, specify it with a colon after the host name:

    <broker_host_IP_address_or_FQDN>:<port_number>
    

  • tls_mode: Parameters for encrypting transmitted data if required, e.g., to comply with the PCI DSS.

    • disabled: Disabled.
    • enabled: Enabled.
      • ca_certificate: CA certificate.

  • subnet_id: ID of the subnet hosting the broker hosts.

• auth: Authentication method used to connect to broker hosts:

  • sasl: SASL authentication.
  • no_auth: Without authentication.

Here is an example of the configuration file structure:

resource "yandex_datatransfer_endpoint" "<endpoint_name_in_Terraform>" {
  name = "<endpoint_name>"
  settings {
    kafka_target {
      security_groups = ["<list_of_security_group_IDs>"]
      connection {
        on_premise {
          broker_urls = ["<list_of_IP_addresses_or_FQDNs_of_broker_hosts>"]
          subnet_id  = "<ID_of_subnet_with_broker_hosts>"
        }
      }
      auth = {
        <authentication_method>
      }
      <topic_settings>
      <serialization_settings>
    }
  }
}

For more information, see the Terraform provider documentation.

• securityGroups: Security groups for network traffic, whose rules will apply to VMs and clusters without changing their settings. For more information, see Networking in Yandex Data Transfer.

• connection.onPremise: Parameters for connecting to the broker hosts:

  • brokerUrls: IP addresses or FQDNs of the broker hosts.

    If the Apache Kafka® port number differs from the standard one, specify it with a colon after the host name:

    <broker_host_IP_address_or_FQDN>:<port_number>
    

  • tlsMode: Parameters for encrypting transmitted data if required, e.g., to comply with the PCI DSS.

    • disabled: Disabled.
    • enabled: Enabled.
      • caCertificate: CA certificate.

  • subnetId: ID of the subnet hosting the broker hosts.

• auth: Authentication method used to connect to broker hosts:

  • sasl: SASL authentication. The following parameters are required:

    • user: Name of the account Data Transfer will use to connect to the topic.
    • password.raw: Password for the account in text form.
    • mechanism: Hashing mechanism.

  • noAuth: Without authentication.

Under topic_settings, specify one of the options to send messages to a topic:

• topic: Specify parameters in this section to send all messages to one topic:

  • topic_name: Name of the topic to send messages to.
  • save_tx_order: Option for saving the order of transactions. Specify true, so as not to split an event stream into independent queues by table.

• topic_prefix: Specify a prefix to send messages to different topics with the given prefix.

  It is similar to the Debezium database.server.name setting. Messages will be sent to a topic named <topic_prefix>.<schema>.<table_name>.

In the topicSettings field, specify one of the options to send messages to a topic:

• topic: Specify parameters in this field to send all messages to one topic:

  • topicName: Name of the topic to send messages to.
  • saveTxOrder: Option for saving the order of transactions. Specify true, so as not to split an event stream into independent queues by table.

• topicPrefix: Specify a prefix to send messages to different topics with the given prefix.

  It is similar to the Debezium database.server.name setting. Messages will be sent to a topic named <topic_prefix>.<schema>.<table_name>.

Under serializer, specify the selected serialization type:

• serializer_auto

• serializer_json

• serializer_debezium

  For this type, you can specify the Debezium serialization parameters as key/value pairs under serializer_debezium.serializer_parameters.

Under serializer, specify the selected serialization type:

• serializerAuto

• serializerJson

• serializerDebezium

  For this type, you can specify the Debezium serialization parameters as key/value pairs under serializerDebezium.serializerParameters.