APIs in Managed Service for Apache Kafka®

A cluster features multiple API types for different operations.

Cloud APIsCloud APIs

Provided by Yandex Cloud, these APIs include the REST API and gRPC API. You can use any of these APIs to manage the following objects:

• Managed Service for Apache Kafka® clusters and cluster logs.
• Some Apache Kafka® entities: topics, accounts, and connectors.

UsageUsage

You do not need to take any preliminary steps to access these APIs as they are always available. To use these APIs, you need to authenticate when sending API requests.

The scope of available operations depends on the roles of the account used to send API requests.

For descriptions of the REST API methods and gRPC API calls, see the API reference. You can find examples of requests in these step-by-step guides.

LimitationsLimitations

You can use these APIs to manage only some of Apache Kafka® entities. For advanced Apache Kafka® entity management, use the Managed Service for Apache Kafka® REST API or Client API.

Managed Service for Apache Kafka® REST APIManaged Service for Apache Kafka® REST API

Provided by Yandex Cloud, these REST APIs include the following:

• REST API for Managed Schema Registry compatible with Confluent Schema Registry API version 6.1 (with some limitations).

  You can use this API to work with the managed schema registry.

• REST API for Apache Kafka® compatible with Confluent REST Proxy API version 6.1 (with some limitations).

  This API enables you to manage producers and consumers using format schemas from the registry as well as view information about brokers, topics, and partitions.

Managed Service for Apache Kafka® clusters use Karapace, an open-source tool which provides access to these REST APIs as well as the basis for the managed schema registry.

UsageUsage

To use any one of these APIs:

1. Get access to the API.

   By default, Managed Schema Registry and both of the REST APIs are disabled and not available. To use them, specify the relevant settings when creating or updating a cluster:

   • Schema registry. Managed Schema Registry and the respective REST API will become available in the cluster.
   • Kafka Rest API. The REST API for Apache Kafka® will become available in the cluster.

   While both of the REST APIs are provided by Karapace, you can access them independently of each other, e.g., only use Managed Schema Registry and its REST API, only use the Apache Kafka® REST API, or all of them at once.

2. Complete all the steps to pre-configure a connection to the cluster.

3. Authenticate when sending API requests.

   The APIs use the [HTTP Basic Authentication]https://developer.mozilla.org/en-US/docs/Web/HTTP/Authentication#basic_authentication_scheme scheme, where each request must contain an Authorization HTTP header with credentials for the Apache Kafka® user sending the request.

The scope of available operations depends on the roles assigned to the Apache Kafka® user sending API requests.

• REST API for Managed Schema Registry:

  • ACCESS_ROLE_CONSUMER role: User can perform read operations with subjects associated with the topic the role was assigned for.
  • ACCESS_ROLE_PRODUCER role: User can perform any operations with subjects associated with the topic the role was assigned for.
  • ACCESS_ROLE_ADMIN role: User can perform any operations with subjects associated with any topic (this role applies to all topics).

  Also, a user’s role determines which subjects this user can manage. For more information about the Managed Schema Registry role model, see Managing data schemas.

• REST API for Apache Kafka®:

  • ACCESS_ROLE_CONSUMER role: User can get information about Apache Kafka® objects and work with consumers.
  • ACCESS_ROLE_PRODUCER or ACCESS_ROLE_ADMIN role: User can perform the full range of operations with Apache Kafka® objects.

For descriptions of the REST API methods for Managed Schema Registry and REST API methods for Apache Kafka®, see the Confluent documentation. For examples of working with these APIs, see Working with the managed data format schema registry via the REST API.

LimitationsLimitations

• REST API for Managed Schema Registry:

  The following limitations apply to registry operation mode management methods:

  • To view information on registry operation modes, the user must have the ACCESS_ROLE_ADMIN permissions.
  • You cannot change the registry operation mode.

• REST API for Apache Kafka®:

  • You cannot manage clusters or other Apache Kafka® objects except producers and consumers.
  • Only the Confluent REST Proxy API v2 methods are available.

Apache Kafka® client APIsApache Kafka® client APIs

Client APIs is an umbrella name for the following native Apache Kafka® APIs:

• Producer API: For operations with producers.
• Consumer API: For operations with consumers.
• Streams API: For operations with data streams.
• Connect API: For operations with connectors.
• Admin API: For operations with Apache Kafka® objects.

You can use these APIs to manage Apache Kafka® objects (with some limitations).

UsageUsage

To use any one of these APIs:

1. Pre-configure your connection to the cluster.

2. Embed the API into your Apache Kafka® client application code.

   To use the client APIs, you need to invoke the relevant functions from the code of your client apps, as these are not REST APIs.

3. To access the APIs, set up the appropriate authentication method in your client app code. Authentication requires the Apache Kafka® user credentials.

   For the descriptions of authentication methods, see the Confluent documentation for client applications. For configuration examples, see Code examples for connecting to an Apache Kafka® cluster.

The scope of available operations depends on the roles assigned to the Apache Kafka® user sending requests to the client APIs.

For descriptions of the client APIs, see the Apache Kafka® documentation For examples of working with the Producer API and Consumer API, see Working with the managed schema registry.

LimitationsLimitations

• You cannot manage Apache Kafka® clusters or accounts.

• You can manage topics via the Admin API with the following limitations:

  • Topics can be managed by an admin user with the ACCESS_ROLE_ADMIN role effective for all the cluster topics. If there is no such user in the cluster, create one.

  • You need to configure user permissions for producers and consumers using native Yandex Cloud interfaces rather than the Apache Kafka® Admin API.