Authenticating and connecting to a database using the Kafka API

EndpointEndpoint

The Kafka API endpoint is displayed in the management console in the Kafka API endpoint field on the Overview tab on the data stream page.

The endpoint has the following format: <FQDN_YDB>:PORT. For example, ydb-01.serverless.yandexcloud.net:9093.

PrerequisitesPrerequisites

To authenticate, take these steps:

1. Create a service account.
2. Assign roles to the service account:
   • For reading from a data stream: ydb.kafkaApi.client and ydb.viewer.
   • For writing to a data stream: ydb.kafkaApi.client and ydb.editor.
3. Create an API key with the yc.ydb.topics.manage scope.

AuthenticationAuthentication

The Kafka API uses the SASL_SSL/PLAIN authentication mechanism.

The following parameters are required:

• <database>: Database path. The database path is displayed in the management console after database= in the Endpoint field on the Overview tab.

  For example, in the grpcs://ydb.serverless.yandexcloud.net:2135/?database=/ru-central1/b1gia87mbaomkfvs6rgl/etnudu2n9ri35luqe4h1 endpoint, /ru-central1/b1gia87mbaomkfvs6rgl/etnudu2n9ri35luqe4h1 is the database path.

• <api-key>: API key.

Parameters used for authentication when reading and writing messages:

• <sasl.username> = @<database> (Note that you need to put @ before the path to the database)
• <sasl.password> = <api-key>

Example of writing and reading a messageExample of writing and reading a message

The example uses the following parameters:

• <kafka-api-endpoint>: Endpoint.
• <stream-name>: Stream name.

1. Install an SSL certificate if you are using a dedicated database:

    sudo mkdir -p /usr/local/share/ca-certificates/Yandex/ && \
    wget "https://storage.yandexcloud.net/cloud-certs/CA.pem" \
     --output-document /usr/local/share/ca-certificates/Yandex/YandexInternalRootCA.crt && \
    sudo chmod 0655 /usr/local/share/ca-certificates/Yandex/YandexInternalRootCA.crt
   

   The certificate will be saved to the /usr/local/share/ca-certificates/Yandex/YandexInternalRootCA.crt file.

2. Install the kcat utility, which is an open source app that can function as a universal data producer or consumer:

   sudo apt-get install kafkacat
   

3. Run this command to get messages from the stream:

   Serverless database

   Dedicated database

   kcat -C \
     -b <kafka-api-endpoint> \
     -t <stream-name> \
     -X security.protocol=SASL_SSL \
     -X sasl.mechanism=PLAIN \
     -X sasl.username="<sasl.username>" \
     -X sasl.password="<sasl.password>"
   

   kcat -C \
     -b <kafka-api-endpoint> \
     -t <stream-name> \
     -X security.protocol=SASL_SSL \
     -X sasl.mechanism=PLAIN \
     -X sasl.username="<sasl.username>" \
     -X sasl.password="<sasl.password>" \
     -X ssl.ca.location=/usr/local/share/ca-certificates/Yandex/YandexInternalRootCA.crt
   

   The command will continuously read new messages from the stream.

4. In a separate terminal, run this command to send a message to the stream:

   Serverless database

   Dedicated database

   echo "test message" | kcat -P \
       -b <kafka-api-endpoint> \
       -t <stream-name> \
       -k key \
       -X security.protocol=SASL_SSL \
       -X sasl.mechanism=PLAIN \
       -X sasl.username="<sasl.username>" \
       -X sasl.password="<sasl.password>"
   

   echo "test message" | kcat -P \
       -b <kafka-api-endpoint> \
       -t <stream-name> \
       -k key \
       -X security.protocol=SASL_SSL \
       -X sasl.mechanism=PLAIN \
       -X sasl.username="<sasl.username>" \
       -X sasl.password="<sasl.password>" \
       -X ssl.ca.location=/usr/local/share/ca-certificates/Yandex/YandexInternalRootCA.crt
   

For core information on how to work with Data Streams using the Kafka API, see the YDB documentation.