Authenticating and connecting to a database using the Kafka API

EndpointEndpoint

The Kafka API endpoint is displayed in the management console on the data stream page under the Overview tab in the Kafka API endpoint field.

The endpoint has the following format: <FQDN_YDB>:PORT. For example, ydb-01.serverless.yandexcloud.net:9093.

PrerequisitesPrerequisites

To authenticate, take these steps:

1. Create a service account.
2. Assign roles to the service account:
   • For reading from a data stream: ydb.kafkaApi.client and ydb.viewer.
   • For writing to a data stream: ydb.kafkaApi.client and ydb.editor.
3. Create an API key.

AuthenticationAuthentication

The Kafka API uses the SASL_SSL/PLAIN authentication mechanism.

The following parameters are required:

• <database>: Database path. To view the database path, in the management console, go to the data stream page and open the Overview tab. The path is specified in the Endpoint field after database=.

  For example, in the grpcs://ydb.serverless.yandexcloud.net:2135/?database=/ru-central1/b1gia87mbaomkfvs6rgl/etnudu2n9ri35luqe4h1 endpoint, /ru-central1/b1gia87mbaomkfvs6rgl/etnudu2n9ri35luqe4h1 is the database path.

• <api-key>: API key.

Parameters used for authentication when reading and writing messages:

• <sasl.username> = @<database>
• <sasl.password> = <api-key>

Example of writing and reading a messageExample of writing and reading a message

The example uses the following parameters:

• <kafka-api-endpoint>: Endpoint
• <stream-name>: Data stream name

1. Install an SSL certificate:

    mkdir -p /usr/local/share/ca-certificates/Yandex/ && \
    wget "https://crls.yandex.net/YandexInternalRootCA.crt" \
     --output-document /usr/local/share/ca-certificates/Yandex/YandexInternalRootCA.crt && \
    chmod 0655 /usr/local/share/ca-certificates/Yandex/YandexInternalRootCA.crt
   

   The certificate will be saved to the /usr/local/share/ca-certificates/Yandex/YandexInternalRootCA.crt file.

2. Install kcat, an open source utility that can function as a generic data producer or consumer:

   sudo apt-get install kafkacat
   

3. Run this command to get messages from the stream:

   kcat -C \
       -b <kafka-api-endpoint> \
       -t <stream-name> \
       -X security.protocol=SASL_SSL \
       -X sasl.mechanism=PLAIN \
       -X sasl.username="<sasl.username>" \
       -X sasl.password="<sasl.password>" \
       -X ssl.ca.location=/usr/local/share/ca-certificates/Yandex/YandexInternalRootCA.crt -Z
   

   The command will continuously read new messages from the stream.

4. In a separate terminal, run this command to send a message to the stream:

   echo "test message" | kcat -P \
       -b <kafka-api-endpoint> \
       -t <stream-name> \
       -k key \
       -X security.protocol=SASL_SSL \
       -X sasl.mechanism=PLAIN \
       -X sasl.username="<sasl.username>" \
       -X sasl.password="<sasl.password>" \
       -X ssl.ca.location=/usr/local/share/ca-certificates/Yandex/YandexInternalRootCA.crt -Z
   

For core information on how to work with Data Streams using the Kafka API, see the YDB documentation.