Managed Service for Apache Kafka® API, gRPC: ConnectorService.Create
- gRPC request
- CreateConnectorRequest
- ConnectorSpec
- ConnectorConfigMirrorMakerSpec
- ClusterConnectionSpec
- ThisClusterSpec
- ExternalClusterConnectionSpec
- ConnectorConfigS3SinkSpec
- S3ConnectionSpec
- ExternalS3StorageSpec
- ConnectorConfigIcebergSinkSpec
- MetastoreConnectionSpec
- IcebergS3ConnectionSpec
- ExternalIcebergS3StorageSpec
- StaticTablesSpec
- DynamicTablesSpec
- IcebergTablesConfigSpec
- IcebergControlSpec
- operation.Operation
- CreateConnectorMetadata
- Connector
- ConnectorConfigMirrorMaker
- ClusterConnection
- ThisCluster
- ExternalClusterConnection
- ConnectorConfigS3Sink
- S3Connection
- ExternalS3Storage
- ConnectorConfigIcebergSink
- MetastoreConnection
- IcebergS3Connection
- ExternalIcebergS3Storage
- StaticTables
- DynamicTables
- IcebergTablesConfig
- IcebergControl
Creates a new Apache Kafka® connector in a cluster.
gRPC request
rpc Create (CreateConnectorRequest) returns (operation.Operation)
CreateConnectorRequest
{
"cluster_id": "string",
"connector_spec": {
"name": "string",
"tasks_max": "google.protobuf.Int64Value",
"properties": "map<string, string>",
// Includes only one of the fields `connector_config_mirrormaker`, `connector_config_s3_sink`, `connector_config_iceberg_sink`
"connector_config_mirrormaker": {
"source_cluster": {
"alias": "string",
// Includes only one of the fields `this_cluster`, `external_cluster`
"this_cluster": "ThisClusterSpec",
"external_cluster": {
"bootstrap_servers": "string",
"sasl_username": "string",
"sasl_password": "string",
"sasl_mechanism": "string",
"security_protocol": "string",
"ssl_truststore_certificates": "string"
}
// end of the list of possible fields
},
"target_cluster": {
"alias": "string",
// Includes only one of the fields `this_cluster`, `external_cluster`
"this_cluster": "ThisClusterSpec",
"external_cluster": {
"bootstrap_servers": "string",
"sasl_username": "string",
"sasl_password": "string",
"sasl_mechanism": "string",
"security_protocol": "string",
"ssl_truststore_certificates": "string"
}
// end of the list of possible fields
},
"topics": "string",
"replication_factor": "google.protobuf.Int64Value"
},
"connector_config_s3_sink": {
"topics": "string",
"file_compression_type": "string",
"file_max_records": "google.protobuf.Int64Value",
"s3_connection": {
"bucket_name": "string",
// Includes only one of the fields `external_s3`
"external_s3": {
"access_key_id": "string",
"secret_access_key": "string",
"endpoint": "string",
"region": "string"
}
// end of the list of possible fields
}
},
"connector_config_iceberg_sink": {
// Includes only one of the fields `topics`, `topics_regex`
"topics": "string",
"topics_regex": "string",
// end of the list of possible fields
"control_topic": "string",
"metastore_connection": {
"catalog_uri": "string",
"warehouse": "string"
},
"s3_connection": {
// Includes only one of the fields `external_s3`
"external_s3": {
"access_key_id": "string",
"secret_access_key": "string",
"endpoint": "string",
"region": "string"
}
// end of the list of possible fields
},
// Includes only one of the fields `static_tables`, `dynamic_tables`
"static_tables": {
"tables": "string"
},
"dynamic_tables": {
"route_field": "string"
},
// end of the list of possible fields
"tables_config": {
"default_commit_branch": "string",
"default_id_columns": "string",
"default_partition_by": "string",
"evolve_schema_enabled": "bool",
"schema_force_optional": "bool",
"schema_case_insensitive": "bool"
},
"control_config": {
"group_id_prefix": "string",
"commit_interval_ms": "google.protobuf.Int64Value",
"commit_timeout_ms": "google.protobuf.Int64Value",
"commit_threads": "google.protobuf.Int64Value",
"transactional_prefix": "string"
}
}
// end of the list of possible fields
}
}
|
Field |
Description |
|
cluster_id |
string Required field. ID of the Apache Kafka® cluster to create the connector in. To get this ID, make a ClusterService.List request. The maximum string length in characters is 50. |
|
connector_spec |
Required field. Configuration of the connector to create. |
ConnectorSpec
An object that represents an Apache Kafka® connector.
See the documentation for details.
|
Field |
Description |
|
name |
string Name of the connector. |
|
tasks_max |
Maximum number of connector tasks. Default value is the number of brokers. |
|
properties |
object (map<string, string>) A set of properties passed to Managed Service for Apache Kafka® with the connector configuration. |
|
connector_config_mirrormaker |
ConnectorConfigMirrorMakerSpec Configuration of the MirrorMaker connector. Includes only one of the fields Additional settings for the connector. |
|
connector_config_s3_sink |
Configuration of S3-Sink connector. Includes only one of the fields Additional settings for the connector. |
|
connector_config_iceberg_sink |
ConnectorConfigIcebergSinkSpec Configuration of Iceberg Sink connector. Includes only one of the fields Additional settings for the connector. |
ConnectorConfigMirrorMakerSpec
|
Field |
Description |
|
source_cluster |
Source cluster configuration for the MirrorMaker connector. |
|
target_cluster |
Target cluster configuration for the MirrorMaker connector. |
|
topics |
string List of Kafka topics, separated by |
|
replication_factor |
Replication factor for automatically created topics. |
ClusterConnectionSpec
|
Field |
Description |
|
alias |
string Alias of cluster connection configuration. |
|
this_cluster |
Connection configuration of the cluster the connector belongs to. As all credentials are already known, leave this parameter empty. Includes only one of the fields Type of connection to Apache Kafka® cluster. |
|
external_cluster |
Configuration of connection to an external cluster with all the necessary credentials. Includes only one of the fields Type of connection to Apache Kafka® cluster. |
ThisClusterSpec
|
Field |
Description |
|
Empty |
|
ExternalClusterConnectionSpec
|
Field |
Description |
|
bootstrap_servers |
string List of bootstrap servers of the cluster, separated by |
|
sasl_username |
string SASL username to use for connection to the cluster. |
|
sasl_password |
string SASL password to use for connection to the cluster. |
|
sasl_mechanism |
string SASL mechanism to use for connection to the cluster. |
|
security_protocol |
string Security protocol to use for connection to the cluster. |
|
ssl_truststore_certificates |
string CA in PEM format to connect to external cluster. |
ConnectorConfigS3SinkSpec
Specification for Kafka S3-Sink Connector.
|
Field |
Description |
|
topics |
string List of Kafka topics, separated by ','. |
|
file_compression_type |
string The compression type used for files put on GCS. |
|
file_max_records |
Max records per file. |
|
s3_connection |
Credentials for connecting to S3 storage. |
S3ConnectionSpec
Specification for S3Connection -
settings of connection to AWS-compatible S3 storage, that
are source or target of Kafka S3-connectors.
YC Object Storage is AWS-compatible.
|
Field |
Description |
|
bucket_name |
string Name of the bucket. |
|
external_s3 |
Configuration for connection to S3 storage. Includes only one of the fields |
ExternalS3StorageSpec
|
Field |
Description |
|
access_key_id |
string ID of the AWS access key. |
|
secret_access_key |
string Secret access key for the AWS access key. |
|
endpoint |
string S3 endpoint. |
|
region |
string AWS region. Default is 'us-east-1'. |
ConnectorConfigIcebergSinkSpec
Specification for Kafka Iceberg Sink Connector.
|
Field |
Description |
|
topics |
string List of Kafka topics, separated by ','. Includes only one of the fields |
|
topics_regex |
string Regex of Kafka topics. Includes only one of the fields |
|
control_topic |
string Control topic name for Iceberg connector. |
|
metastore_connection |
Credentials for connecting to Managed Hive Metastore. |
|
s3_connection |
Credentials for connecting to S3 storage. |
|
static_tables |
Static table routing Includes only one of the fields Table routing strategy |
|
dynamic_tables |
Dynamic table routing Includes only one of the fields Table routing strategy |
|
tables_config |
Optional table settings |
|
control_config |
Optional control settings |
MetastoreConnectionSpec
|
Field |
Description |
|
catalog_uri |
string Thrift URI of Hive Metastore |
|
warehouse |
string Warehouse root directory in S3 |
IcebergS3ConnectionSpec
Specification for IcebergS3Connection -
settings of connection to AWS-compatible S3 storage, that
are target of Kafka Iceberg-connectors.
YC Object Storage is AWS-compatible.
|
Field |
Description |
|
external_s3 |
Configuration for connection to S3 storage. Includes only one of the fields |
ExternalIcebergS3StorageSpec
|
Field |
Description |
|
access_key_id |
string ID of the AWS access key. |
|
secret_access_key |
string Secret access key for the AWS access key. |
|
endpoint |
string S3 endpoint. |
|
region |
string AWS region. Default is 'us-east-1'. |
StaticTablesSpec
|
Field |
Description |
|
tables |
string List of tables, separated by ','. |
DynamicTablesSpec
|
Field |
Description |
|
route_field |
string Field in the message to define the target table |
IcebergTablesConfigSpec
|
Field |
Description |
|
default_commit_branch |
string Default Git-like branch name for Iceberg commits. |
|
default_id_columns |
string List of columns used as identifiers for upsert operations, separated by ','. |
|
default_partition_by |
string Comma-separated list of columns or transform expressions for table partitioning.
|
|
evolve_schema_enabled |
bool Enable automatic schema evolution. |
|
schema_force_optional |
bool Force all columns to be nullable (optional). |
|
schema_case_insensitive |
bool Enable case-insensitive field name matching. |
IcebergControlSpec
|
Field |
Description |
|
group_id_prefix |
string Consumer group ID prefix for control topic. |
|
commit_interval_ms |
Interval between commits in milliseconds. |
|
commit_timeout_ms |
Commit operation timeout in milliseconds. |
|
commit_threads |
Number of threads for commit operations. |
|
transactional_prefix |
string Prefix for transactional operations. |
operation.Operation
{
"id": "string",
"description": "string",
"created_at": "google.protobuf.Timestamp",
"created_by": "string",
"modified_at": "google.protobuf.Timestamp",
"done": "bool",
"metadata": {
"cluster_id": "string",
"connector_name": "string"
},
// Includes only one of the fields `error`, `response`
"error": "google.rpc.Status",
"response": {
"name": "string",
"tasks_max": "google.protobuf.Int64Value",
"properties": "map<string, string>",
"health": "Health",
"status": "Status",
"cluster_id": "string",
// Includes only one of the fields `connector_config_mirrormaker`, `connector_config_s3_sink`, `connector_config_iceberg_sink`
"connector_config_mirrormaker": {
"source_cluster": {
"alias": "string",
// Includes only one of the fields `this_cluster`, `external_cluster`
"this_cluster": "ThisCluster",
"external_cluster": {
"bootstrap_servers": "string",
"sasl_username": "string",
"sasl_mechanism": "string",
"security_protocol": "string"
}
// end of the list of possible fields
},
"target_cluster": {
"alias": "string",
// Includes only one of the fields `this_cluster`, `external_cluster`
"this_cluster": "ThisCluster",
"external_cluster": {
"bootstrap_servers": "string",
"sasl_username": "string",
"sasl_mechanism": "string",
"security_protocol": "string"
}
// end of the list of possible fields
},
"topics": "string",
"replication_factor": "google.protobuf.Int64Value"
},
"connector_config_s3_sink": {
"topics": "string",
"file_compression_type": "string",
"file_max_records": "google.protobuf.Int64Value",
"s3_connection": {
"bucket_name": "string",
// Includes only one of the fields `external_s3`
"external_s3": {
"access_key_id": "string",
"endpoint": "string",
"region": "string"
}
// end of the list of possible fields
}
},
"connector_config_iceberg_sink": {
// Includes only one of the fields `topics`, `topics_regex`
"topics": "string",
"topics_regex": "string",
// end of the list of possible fields
"control_topic": "string",
"metastore_connection": {
"catalog_uri": "string",
"warehouse": "string"
},
"s3_connection": {
// Includes only one of the fields `external_s3`
"external_s3": {
"access_key_id": "string",
"endpoint": "string",
"region": "string"
}
// end of the list of possible fields
},
// Includes only one of the fields `static_tables`, `dynamic_tables`
"static_tables": {
"tables": "string"
},
"dynamic_tables": {
"route_field": "string"
},
// end of the list of possible fields
"tables_config": {
"default_commit_branch": "string",
"default_id_columns": "string",
"default_partition_by": "string",
"evolve_schema_enabled": "bool",
"schema_force_optional": "bool",
"schema_case_insensitive": "bool"
},
"control_config": {
"group_id_prefix": "string",
"commit_interval_ms": "google.protobuf.Int64Value",
"commit_timeout_ms": "google.protobuf.Int64Value",
"commit_threads": "google.protobuf.Int64Value",
"transactional_prefix": "string"
}
}
// end of the list of possible fields
}
// end of the list of possible fields
}
An Operation resource. For more information, see Operation.
|
Field |
Description |
|
id |
string ID of the operation. |
|
description |
string Description of the operation. 0-256 characters long. |
|
created_at |
Creation timestamp. |
|
created_by |
string ID of the user or service account who initiated the operation. |
|
modified_at |
The time when the Operation resource was last modified. |
|
done |
bool If the value is |
|
metadata |
Service-specific metadata associated with the operation. |
|
error |
The error result of the operation in case of failure or cancellation. Includes only one of the fields The operation result. |
|
response |
The normal response of the operation in case of success. Includes only one of the fields The operation result. |
CreateConnectorMetadata
|
Field |
Description |
|
cluster_id |
string ID of the Apache Kafka® cluster the connector is being created in. |
|
connector_name |
string Required field. Name of the Apache Kafka® connector that is being created. The maximum string length in characters is 256. Value must match the regular expression |
Connector
|
Field |
Description |
|
name |
string Name of the connector. |
|
tasks_max |
Maximum number of connector tasks. Default value is the number of brokers. |
|
properties |
object (map<string, string>) A set of properties passed to Managed Service for Apache Kafka® with the connector configuration. |
|
health |
enum Health Connector health.
|
|
status |
enum Status Current status of the connector.
|
|
cluster_id |
string ID of the Apache Kafka® cluster that the connector belongs to. |
|
connector_config_mirrormaker |
Configuration of the MirrorMaker connector. Includes only one of the fields Additional settings for the connector. |
|
connector_config_s3_sink |
Configuration of S3-Sink connector. Includes only one of the fields Additional settings for the connector. |
|
connector_config_iceberg_sink |
Configuration of Iceberg Sink connector. Includes only one of the fields Additional settings for the connector. |
ConnectorConfigMirrorMaker
|
Field |
Description |
|
source_cluster |
Source cluster connection configuration. |
|
target_cluster |
Target cluster connection configuration. |
|
topics |
string List of Kafka topics, separated by |
|
replication_factor |
Replication factor for automatically created topics. |
ClusterConnection
|
Field |
Description |
|
alias |
string Alias of cluster connection configuration. |
|
this_cluster |
Connection configuration of the cluster the connector belongs to. As all credentials are already known, leave this parameter empty. Includes only one of the fields Type of connection to Apache Kafka® cluster. |
|
external_cluster |
Configuration of connection to an external cluster with all the necessary credentials. Includes only one of the fields Type of connection to Apache Kafka® cluster. |
ThisCluster
|
Field |
Description |
|
Empty |
|
ExternalClusterConnection
|
Field |
Description |
|
bootstrap_servers |
string List of bootstrap servers of the cluster, separated by |
|
sasl_username |
string SASL username to use for connection to the cluster. |
|
sasl_mechanism |
string SASL mechanism to use for connection to the cluster. |
|
security_protocol |
string Security protocol to use for connection to the cluster. |
ConnectorConfigS3Sink
An Apache Kafka® S3-Sink
connector resource.
|
Field |
Description |
|
topics |
string List of Kafka topics, separated by ','. |
|
file_compression_type |
string The compression type used for files put on GCS. |
|
file_max_records |
Max records per file. |
|
s3_connection |
Credentials for connecting to S3 storage. |
S3Connection
Resource for S3Connection -
settings of connection to AWS-compatible S3 storage, that
are source or target of Kafka S3-connectors.
YC Object Storage is AWS-compatible.
|
Field |
Description |
|
bucket_name |
string Name of the bucket. |
|
external_s3 |
Configuration for connection to S3 storage. Includes only one of the fields |
ExternalS3Storage
|
Field |
Description |
|
access_key_id |
string ID of the AWS access key. |
|
endpoint |
string S3 endpoint. |
|
region |
string AWS region. Default is 'us-east-1'. |
ConnectorConfigIcebergSink
Resource for Kafka Iceberg Sink Connector.
|
Field |
Description |
|
topics |
string List of Kafka topics, separated by ','. Includes only one of the fields |
|
topics_regex |
string Regex of Kafka topics. Includes only one of the fields |
|
control_topic |
string Control topic name for Iceberg connector. |
|
metastore_connection |
Credentials for connecting to Managed Hive Metastore. |
|
s3_connection |
Credentials for connecting to S3 storage. |
|
static_tables |
Static table routing Includes only one of the fields Table routing strategy |
|
dynamic_tables |
Dynamic table routing Includes only one of the fields Table routing strategy |
|
tables_config |
Optional table settings |
|
control_config |
Optional control settings |
MetastoreConnection
|
Field |
Description |
|
catalog_uri |
string Thrift URI of Hive Metastore |
|
warehouse |
string Warehouse root directory in S3 |
IcebergS3Connection
Resource for IcebergS3Connection -
settings of connection to AWS-compatible S3 storage, that
are target of Kafka Iceberg-connectors.
YC Object Storage is AWS-compatible.
|
Field |
Description |
|
external_s3 |
Configuration for connection to S3 storage. Includes only one of the fields |
ExternalIcebergS3Storage
|
Field |
Description |
|
access_key_id |
string ID of the AWS access key. |
|
endpoint |
string S3 endpoint. |
|
region |
string AWS region. Default is 'us-east-1'. |
StaticTables
|
Field |
Description |
|
tables |
string List of tables, separated by ','. |
DynamicTables
|
Field |
Description |
|
route_field |
string Field in the message to define the target table |
IcebergTablesConfig
|
Field |
Description |
|
default_commit_branch |
string Default Git-like branch name for Iceberg commits. |
|
default_id_columns |
string List of columns used as identifiers for upsert operations, separated by ','. |
|
default_partition_by |
string Comma-separated list of columns or transform expressions for table partitioning.
|
|
evolve_schema_enabled |
bool Enable automatic schema evolution. |
|
schema_force_optional |
bool Force all columns to be nullable (optional). |
|
schema_case_insensitive |
bool Enable case-insensitive field name matching. |
IcebergControl
|
Field |
Description |
|
group_id_prefix |
string Consumer group ID prefix for control topic. |
|
commit_interval_ms |
Interval between commits in milliseconds. |
|
commit_timeout_ms |
Commit operation timeout in milliseconds. |
|
commit_threads |
Number of threads for commit operations. |
|
transactional_prefix |
string Prefix for transactional operations. |