Managing connectors

If you do not have the Yandex Cloud command line interface yet, install and initialize it.

The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

To get a list of cluster connectors, run the command:

yc managed-kafka connector list --cluster-name=<cluster_name>

Result:

+--------------+-----------+
|     NAME     | TASKS MAX |
+--------------+-----------+
| connector559 |         1 |
| ...          |           |
+--------------+-----------+

You can retrieve the cluster name with a list of clusters in the folder.

To get a list of connectors, use the list REST API method for the Connector resource or the ConnectorService/List gRPC API call and provide the cluster ID in the clusterId request parameter.

To find out the cluster ID, get a list of clusters in the folder.

If you do not have the Yandex Cloud command line interface yet, install and initialize it.

The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

To get detailed information about a connector, run this command:

yc managed-kafka connector get <connector_name>\
   --cluster-name=<cluster_name>

Result:

name: connector785
tasks_max: "1"
cluster_id: c9qbkmoiimsl********
...

You can request the connector name with a list of cluster connectors and the cluster name with a list of clusters in the folder.

To get connector details, use the get REST API method for the Connector resource or the ConnectorService/Get gRPC API call and provide the following in the request:

• Cluster ID in the clusterId parameter. To find out the cluster ID, get a list of clusters in the folder.
• Connector name in the connectorName parameter. To find out the name, retrieve a list of cluster connectors.

If you do not have the Yandex Cloud command line interface yet, install and initialize it.

The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

To create a MirrorMaker connector:

1. View the description of the CLI command to create a connector:

   yc managed-kafka connector-mirrormaker create --help
   

2. Create a connector:

   yc managed-kafka connector-mirrormaker create <connector_name> \
      --cluster-name=<cluster_name> \
      --direction=<connector_direction> \
      --tasks-max=<task_limit> \
      --properties=<advanced_properties> \
      --replication-factor=<replication_factor> \
      --topics=<topics_template> \
      --this-cluster-alias=<this_cluster_prefix> \
      --external-cluster alias=<external_cluster_prefix>,`
                        `bootstrap-servers=<list_of_broker_host_FQDNs>,`
                        `security-protocol=<security_protocol>,`
                        `sasl-mechanism=<encryption_mechanism>,`
                        `sasl-username=<username>,`
                        `sasl-password=<user_password>,`
                        `ssl-truststore-certificates=<certificates_in_PEM_format>
   

   For info on how to get a broker host's FQDN, see this guide.

   You can retrieve the cluster name with a list of clusters in the folder.

   --direction takes these values:

   • egress: If the current cluster is a source cluster.
   • ingress: If the current cluster is a target cluster.

To create an S3 Sink connector:

1. View the description of the CLI command to create a connector:

   yc managed-kafka connector-s3-sink create --help
   

2. Create a connector:

   yc managed-kafka connector-s3-sink create <connector_name> \
      --cluster-name=<cluster_name> \
      --tasks-max=<task_limit> \
      --properties=<advanced_properties> \
      --topics=<topics_template> \
      --file-compression-type=<compression_codec> \
      --file-max-records=<file_max_records> \
      --bucket-name=<bucket_name> \
      --access-key-id=<AWS_compatible_static_key_ID> \
      --secret-access-key=<AWS_compatible_static_key_contents> \
      --storage-endpoint=<S3_compatible_storage_endpoint> \
      --region=<S3_compatible_storage_region>
   

   You can retrieve the cluster name with a list of clusters in the folder.

1. Check the list of MirrorMaker and S3 Sink connector settings.

2. Open the current Terraform configuration file with an infrastructure plan.

   For more information about creating this file, see Creating clusters.

3. To create a MirrorMaker connector, add the yandex_mdb_kafka_connector resource with the connector_config_mirrormaker settings section:

   resource "yandex_mdb_kafka_connector" "<connector_name>" {
     cluster_id = "<cluster_ID>"
     name       = "<connector_name>"
     tasks_max  = <task_limit>
     properties = {
       <advanced_properties>
     }
     connector_config_mirrormaker {
       topics             = "<topics_template>"
       replication_factor = <replication_factor>
       source_cluster {
         alias = "<cluster_prefix>"
         external_cluster {
           bootstrap_servers           = "<list_of_broker_host_FQDNs>"
           sasl_username               = "<username>"
           sasl_password               = "<user_password>"
           sasl_mechanism              = "<encryption_mechanism>"
           security_protocol           = "<security_protocol>"
           ssl-truststore-certificates = "<PEM_certificate_contents>"
         }
       }
       target_cluster {
         alias = "<cluster_prefix>"
         this_cluster {}
       }
     }
   }
   

   For info on how to get a broker host's FQDN, see this guide.

4. To create an S3 Sink connector, add the yandex_mdb_kafka_connector resource with the connector_config_s3_sink settings section:

   resource "yandex_mdb_kafka_connector" "<connector_name>" {
     cluster_id = "<cluster_ID>"
     name       = "<connector_name>"
     tasks_max  = <task_limit>
     properties = {
       <advanced_properties>
     }
     connector_config_s3_sink {
       topics                = "<topics_template>"
       file_compression_type = "<compression_codec>"
       file_max_records      = <file_max_records>
       s3_connection {
         bucket_name = "<bucket_name>"
         external_s3 {
           endpoint          = "<S3_compatible_storage_endpoint>"
           access_key_id     = "<AWS_compatible_static_key_ID>"
           secret_access_key = "<AWS_compatible_static_key_contents>"
         }
       }
     }
   }
   

5. Make sure the settings are correct.

   1. Using the command line, navigate to the folder that contains the up-to-date Terraform configuration files with an infrastructure plan.

   2. Run the command:

      terraform validate
      

      If there are errors in the configuration files, Terraform will point to them.

6. Confirm updating the resources.

   1. Run the command to view planned changes:

      terraform plan
      

      If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.

   2. If you are happy with the planned changes, apply them:

      1. Run the command:

         terraform apply
         

      2. Confirm the update of resources.

      3. Wait for the operation to complete.

For more information, see the Terraform provider documentation.

To create a connector, use the create REST API method for the Connector resource or the ConnectorService/Create gRPC API call and provide the following in the request:

• ID of the cluster you want to create a connector in, in the clusterId parameter. To find out the cluster ID, get a list of clusters in the folder.
• MirrorMaker or S3 Sink connector settings in the connectorSpec parameter.

• --cluster-name: Cluster name.

• --direction: Connector direction:

  • ingress: For a target cluster.
  • egress: For a source cluster.

• --tasks-max: Number of concurrent processes. To distribute replication load evenly, we recommend a value of at least 2.

• --properties: Comma-separated list of advanced connector settings in <key>:<value> format. Here are some sample keys:

  • key.converter
  • value.converter

  For the list of common connector settings, see the Apache Kafka® documentation.

• --replication-factor: Number of topic copies stored in the cluster.

• --topics: Template for selecting topics to replicate. Topic names in the list are separated by a comma or |. You may use the .* expression, e.g., analysis.*. To migrate all topics, put .*.

• --this-cluster-alias: This cluster prefix in the connector settings.

• --external-cluster: External cluster parameters:

  • alias: External cluster prefix in the connector settings.

  • bootstrap-servers: Comma-separated list of the FQDNs of the external cluster broker hosts with the port numbers for connection.

    For info on how to get a broker host's FQDN, see this guide.

  • security-protocol: Connector connection protocol:

    • plaintext, sasl_plaintext: To connect without SSL.
    • ssl, sasl_ssl: To connect with SSL.

  • sasl-mechanism: Username and password encryption mechanism.

  • sasl-username: Username for connecting the connector to the external cluster.

  • sasl-password: User password for connecting the connector to the external cluster.

  • ssl-truststore-certificates: List of PEM certificates.

• properties: Comma-separated list of advanced connector settings in <key>:<value> format. Here are some sample keys:

  • key.converter
  • value.converter

  For the list of common connector settings, see the Apache Kafka® documentation.

• topics: Template for selecting topics to replicate. Topic names in the list are separated by a comma or |. You may use the .* expression, e.g., analysis.*. To migrate all topics, put .*.

• replication_factor: Number of topic copies stored in the cluster.

• source_cluster and target_cluster: Parameters for connecting to the source cluster and target cluster:

  • alias: Cluster prefix in the connector settings.

    Note

    Topics in the target cluster will be created with the specified prefix.

  • this_cluster: Option to use the current cluster as a source or target.

  • external_cluster: Parameters for connecting to the external cluster:

    • bootstrap_servers: Comma-separated list of the FQDNs of the cluster broker hosts with the port numbers for connection.

      For info on how to get a broker host's FQDN, see this guide.

    • sasl_username: Username for connecting the connector to the cluster.

    • sasl_password: User password for connecting the connector to the cluster.

    • sasl_mechanism: Username and password encryption mechanism.

    • security_protocol: Connector connection protocol:

      • PLAINTEXT, SASL_PLAINTEXT: To connect without SSL.
      • SSL, SASL_SSL: To connect with SSL.

    • ssl_truststore_certificates: PEM certificate contents.

MirrorMaker connector settings are specified in the connectorConfigMirrormaker (connector_config_mirrormaker) parameter:

• sourceCluster (source_cluster) and targetCluster (target_cluster): Parameters for connecting to the source cluster and target cluster:

  • alias: Cluster prefix in the connector settings.

    Note

    Topics in the target cluster will be created with the specified prefix.

  • thisCluster (this_cluster): Option to use the current cluster as a source or target.

  • externalCluster (external_cluster): Parameters for connecting to the external cluster:

    • bootstrapServers (bootstrap_servers): Comma-separated list of the FQDNs of the cluster broker hosts with the port numbers for connection.

      For info on how to get a broker host's FQDN, see this guide.

    • saslUsername (sasl_username): Username for connecting the connector to the cluster.

    • saslPassword (sasl_password): User password for connecting the connector to the cluster.

    • saslMechanism (sasl_mechanism): Username and password encryption mechanism.

    • securityProtocol (security_protocol): Connector connection protocol:

      • PLAINTEXT, SASL_PLAINTEXT: To connect without SSL.
      • SSL, SASL_SSL: To connect with SSL.

    • sslTruststoreCertificates (ssl_truststore_certificates): PEM certificate contents.

• topics: Template for selecting topics to replicate. Topic names in the list are separated by a comma or |. You may use the .* expression, e.g., analysis.*. To migrate all topics, put .*.

• replicationFactor (replication_factor): Number of topic copies stored in the cluster.

• --cluster-name: Cluster name.

• --tasks-max: Number of concurrent processes. To distribute replication load evenly, we recommend a value of at least 2.

• --properties: Comma-separated list of advanced connector settings in <key>:<value> format. Here are some sample keys:

  • key.converter
  • value.converter
  • value.converter.schemas.enable
  • format.output.type

  For the list of all connector settings, see the connector documentation. For the list of common connector settings, see the Apache Kafka® documentation.

• --topics: Template for selecting topics to replicate. Topic names in the list are separated by a comma or |. You may use the .* expression, e.g., analysis.*. To migrate all topics, put .*.

• --file-compression-type: Message compression codec. You cannot change this parameter after creating the cluster. Acceptable values:

  • none (default): No compression
  • gzip: gzip codec
  • snappy: snappy codec
  • zstd: zstd codec

• --file-max-records: Maximum number of records that can be written to a single file in an S3-compatible storage.

• --bucket-name: Name of the S3-compatible storage bucket to write data to.

• --storage-endpoint: Endpoint for storage access (to be requested from the storage provider). Example: storage.yandexcloud.net.

• --region: Region where the S3-compatible storage bucket is located. The default value is us-east-1. Available regions.

• --access-key-id, --secret-access-key: AWS-compatible key ID and contents.

• properties: Comma-separated list of advanced connector settings in <key>:<value> format. Here are some sample keys:

  • key.converter
  • value.converter
  • value.converter.schemas.enable
  • format.output.type

  For the list of all connector settings, see the connector documentation. For the list of common connector settings, see the Apache Kafka® documentation.

• topics: Template for selecting topics to replicate. Topic names in the list are separated by a comma or |. You may use the .* expression, e.g., analysis.*. To migrate all topics, put .*.

• file_compression_type: Codec for message compression. You cannot change this parameter after creating the cluster. Acceptable values:

  • none (default): No compression
  • gzip: gzip codec
  • snappy: snappy codec
  • zstd: zstd codec

• file_max_records: Maximum number of records that can be written to a single file in an S3-compatible storage.

• s3_connection: S3-compatible storage connection parameters:

  • bucket_name: Name of the bucket to write data to.

  • external_s3: External S3-compatible storage connection parameters:

    • endpoint: Endpoint for storage access (to find out from storage provider). Example: storage.yandexcloud.net.

    • region: Region where the S3-compatible storage bucket is located. The default value is us-east-1. Available regions.

    • access_key_id, secret_access_key: AWS-compatible key ID and contents.

The S3 Sink connector settings are specified in the connectorConfigS3Sink (connector_config_s3_sink) parameter:

• topics: Template for selecting topics to replicate. Topic names in the list are separated by a comma or |. You may use the .* expression, e.g., analysis.*. To migrate all topics, put .*.

• fileCompressionType (file_compression_type): Message compression codec. You cannot change this parameter after creating the cluster. Acceptable values:

  • none (default): No compression
  • gzip: gzip codec
  • snappy: snappy codec
  • zstd: zstd codec

• fileMaxRecords (file_max_records): Maximum number of records that can be written to a single file in an S3-compatible storage.

• s3Connection (s3_connection): S3-compatible storage connection parameters:

  • bucketName (bucket_name): Name of the bucket to write data to.
  • externalS3 (external_s3): External storage parameters:

    • endpoint: Endpoint for storage access (to be requested from the storage provider) Example: storage.yandexcloud.net.

    • region: Region where the S3-compatible storage bucket is located. The default value is us-east-1. Available regions.

    • accessKeyId (access_key_id), secretAccessKey (secret_access_key): AWS-compatible key ID and contents.

If you do not have the Yandex Cloud command line interface yet, install and initialize it.

The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

To edit a MirrorMaker connector:

1. View the description of the CLI command to edit a connector:

   yc managed-kafka connector-mirrormaker update --help
   

2. Run an operation, e.g., the task limit update operation:

   yc managed-kafka connector-mirrormaker update <connector_name> \
      --cluster-name=<cluster_name> \
      --direction=<connector_direction> \
      --tasks-max=<new_task_limit>
   

   Where --direction is the connector direction: ingress or egres.

   You can request the connector name with a list of cluster connectors and the cluster name with a list of clusters in the folder.

To update the S3 Sink connector:

1. View the description of the CLI command to edit a connector:

   yc managed-kafka connector-s3-sink update --help
   

2. Run an operation, e.g., the task limit update operation:

   yc managed-kafka connector-s3-sink update <connector_name> \
      --cluster-name=<cluster_name> \
      --tasks-max=<new_task_limit>
   

   You can request the connector name with a list of cluster connectors and the cluster name with a list of clusters in the folder.

1. Check the list of MirrorMaker and S3 Sink connector settings.

2. Open the current Terraform configuration file with an infrastructure plan.

   For more information about creating this file, see Creating clusters.

3. Edit the parameter values in the yandex_mdb_kafka_connector resource description:

   • For a MirrorMaker connector:

     resource "yandex_mdb_kafka_connector" "<connector_name>" {
       cluster_id = "<cluster_ID>"
       name       = "<connector_name>"
       tasks_max  = <task_limit>
       properties = {
         <advanced_properties>
       }
       connector_config_mirrormaker {
         topics             = "<topics_template>"
         replication_factor = <replication_factor>
         source_cluster {
           alias = "<cluster_prefix>"
           external_cluster {
             bootstrap_servers           = "<list_of_broker_host_FQDNs>"
             sasl_username               = "<username>"
             sasl_password               = "<user_password>"
             sasl_mechanism              = "<encryption_mechanism>"
             security_protocol           = "<security_protocol>"
             ssl-truststore-certificates = "<PEM_certificate_contents>"
           }
         }
         target_cluster {
           alias = "<cluster_prefix>"
           this_cluster {}
         }
       }
     }
     

   • For the S3 Sink connector:

     resource "yandex_mdb_kafka_connector" "<S3_Sink_connector_name>" {
       cluster_id = "<cluster_ID>"
       name       = "<S3_Sink_connector_name>"
       tasks_max  = <task_limit>
       properties = {
         <advanced_properties>
      }
       connector_config_s3_sink {
         topics                = "<topics_template>"
         file_max_records      = <file_max_records>
         s3_connection {
           bucket_name = "<bucket_name>"
           external_s3 {
             endpoint          = "<S3_compatible_storage_endpoint>"
             access_key_id     = "<AWS_compatible_static_key_ID>"
             secret_access_key = "<AWS_compatible_static_key_contents>"
           }
         }
       }
     }
     

4. Make sure the settings are correct.

   1. Using the command line, navigate to the folder that contains the up-to-date Terraform configuration files with an infrastructure plan.

   2. Run the command:

      terraform validate
      

      If there are errors in the configuration files, Terraform will point to them.

5. Confirm updating the resources.

   1. Run the command to view planned changes:

      terraform plan
      

      If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.

   2. If you are happy with the planned changes, apply them:

      1. Run the command:

         terraform apply
         

      2. Confirm the update of resources.

      3. Wait for the operation to complete.

For more information, see the Terraform provider documentation.

To update a connector, use the update REST API method for the Connector resource or the ConnectorService/Update gRPC API call and provide the following in the request:

• ID of the cluster you want to update a connector in, in the clusterId parameter. To find out the cluster ID, get a list of clusters in the folder.
• MirrorMaker or S3 Sink connector settings in the connectorSpec parameter.

If you do not have the Yandex Cloud command line interface yet, install and initialize it.

The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

To pause a connector, run the command:

yc managed-kafka connector pause <connector_name> \
   --cluster-name=<cluster_name>

To pause a connector, use the pause REST API method for the Connector resource or the ConnectorService/Pause gRPC API call and provide the following in the request:

• Cluster ID in the clusterId parameter. To find out the cluster ID, get a list of clusters in the folder.
• Connector name in the connectorName parameter. To find out the name, retrieve a list of cluster connectors.

If you do not have the Yandex Cloud command line interface yet, install and initialize it.

The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

To resume a connector, run the command:

yc managed-kafka connector resume <connector_name> \
   --cluster-name=<cluster_name>

To resume a connector, use the resume REST API method for the Connector resource or the ConnectorService/Resume gRPC API call and provide the following in the request:

• Cluster ID in the clusterId parameter. To find out the cluster ID, get a list of clusters in the folder.
• Connector name in the connectorName parameter. To find out the name, retrieve a list of cluster connectors.

If you do not have the Yandex Cloud command line interface yet, install and initialize it.

The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

To delete a connector, run the command:

yc managed-kafka connector delete <connector_name> \
   --cluster-name <cluster_name>

1. Open the current Terraform configuration file with an infrastructure plan.

   For more information about creating this file, see Creating clusters.

2. Delete the yandex_mdb_kafka_connector resource with the connector description.

3. Make sure the settings are correct.

   1. Using the command line, navigate to the folder that contains the up-to-date Terraform configuration files with an infrastructure plan.

   2. Run the command:

      terraform validate
      

      If there are errors in the configuration files, Terraform will point to them.

4. Confirm updating the resources.

   1. Run the command to view planned changes:

      terraform plan
      

      If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.

   2. If you are happy with the planned changes, apply them:

      1. Run the command:

         terraform apply
         

      2. Confirm the update of resources.

      3. Wait for the operation to complete.

For more information, see the Terraform provider documentation.

To delete a connector, use the delete REST API method for the Connector resource or the ConnectorService/Delete gRPC API call and provide the following in the request:

• Cluster ID in the clusterId parameter. To find out the cluster ID, get a list of clusters in the folder.
• Connector name in the connectorName parameter. To find out the name, retrieve a list of cluster connectors.