Authenticating and connecting to a database using the Kafka API

EndpointEndpoint

The Kafka API endpoint is displayed in the management console on the data stream page under the Overview tab in the Kafka API endpoint field.

The endpoint has the following format: <FQDN_YDB>:PORT, e.g., ydb-01.serverless.yandexcloud.net:9093.

PrerequisitesPrerequisites

To authenticate, take these steps:

1. Create a service account.
2. Assign roles to the service account:
   • ydb.kafkaApi.client and ydb.viewer to read data from the data stream.
   • ydb.kafkaApi.client and ydb.editor to write data to the data stream.
3. Create an API key with the yc.ydb.topics.manage scope.

AuthenticationAuthentication

The Kafka API uses the SASL_SSL/PLAIN authentication mechanism.

The following parameters are required:

• <database>: Database path. To view the database path, in the management console, go to the data stream page and open the Overview tab. The path is specified in the Endpoint field after database=.

  For example, in the grpcs://ydb.serverless.yandexcloud.net:2135/?database=/ru-central1/b1gia87mbaomkfvs6rgl/etnudu2n9ri35luqe4h1 endpoint, /ru-central1/b1gia87mbaomkfvs6rgl/etnudu2n9ri35luqe4h1 is the database path.

• <api-key>: API key.

Parameters used for authentication when reading and writing messages:

• <sasl.username> = @<database> (Note that you need to add @ before the database path.)
• <sasl.password> = <api-key>

Example of writing and reading a messageExample of writing and reading a message

The example uses the following parameters:

• <kafka-api-endpoint>: Endpoint.
• <stream-name>: Stream name.

1. Install an SSL certificate:

    mkdir -p /usr/local/share/ca-certificates/Yandex/ && \
    wget "https://crls.yandex.net/YandexInternalRootCA.crt" \
     --output-document /usr/local/share/ca-certificates/Yandex/YandexInternalRootCA.crt && \
    chmod 0655 /usr/local/share/ca-certificates/Yandex/YandexInternalRootCA.crt
   

   The certificate will be saved to the /usr/local/share/ca-certificates/Yandex/YandexInternalRootCA.crt file.

2. Install the kcat utility, which is an open source app that can function as a universal data producer or consumer:

   sudo apt-get install kafkacat
   

3. Run this command to get messages from the stream:

   kcat -C \
       -b <kafka-api-endpoint> \
       -t <stream-name> \
       -X security.protocol=SASL_SSL \
       -X sasl.mechanism=PLAIN \
       -X sasl.username="<sasl.username>" \
       -X sasl.password="<sasl.password>" \
       -X ssl.ca.location=/usr/local/share/ca-certificates/Yandex/YandexInternalRootCA.crt -Z
   

   The command will continuously read new messages from the stream.

4. In a separate terminal, run this command to send a message to the stream:

   echo "test message" | kcat -P \
       -b <kafka-api-endpoint> \
       -t <stream-name> \
       -k key \
       -X security.protocol=SASL_SSL \
       -X sasl.mechanism=PLAIN \
       -X sasl.username="<sasl.username>" \
       -X sasl.password="<sasl.password>" \
       -X ssl.ca.location=/usr/local/share/ca-certificates/Yandex/YandexInternalRootCA.crt -Z
   

For core information on how to work with Data Streams using the Kafka API, see the YDB documentation.