Viewing Apache Kafka® cluster logs

If you do not have the Yandex Cloud (CLI) command line interface yet, install and initialize it.

The folder specified when creating the CLI profile is used by default. To change the default folder, use the yc config set folder-id <folder_ID> command. You can specify a different folder using the --folder-name or --folder-id parameter.

1. View the description of the CLI command to view cluster logs:

   yc managed-kafka cluster list-logs --help
   

2. Run the following command to get cluster logs (our example does not contain a complete list of available parameters):

   yc managed-kafka cluster list-logs <cluster_name_or_ID> \
      --limit <entry_number_limit> \
      --columns <log_columns_list> \
      --filter <entry_filtration_settings> \
      --since <time_range_left_boundary> \
      --until <time_range_right_boundary>
   

   Where:

   • --limit: limits on the number of entries to output.

   • --columns: List of log columns to draw data from.

     • hostname: Host name.
     • message: Message output by the component.
     • severity: Logging level. Output example: INFO.
     • origin: Message origin. Output examples: kafka_server or kafka_controller.

   • --filter: record filter settings, for example, message.hostname='node1.mdb.yandexcloud.net'.

   • --since: Left boundary of a time range in RFC-3339, HH:MM:SS format or a time interval relative to the current time. Examples: 2006-01-02T15:04:05Z, 15:04:05, 2h, 3h30m ago.

   • --until: right boundary of a time range, the format is similar to that of --since.

You can request the cluster name and ID with the list of clusters in the folder.

1. Get an IAM token for API authentication and put it into the environment variable:

   export IAM_TOKEN="<IAM_token>"
   

2. Use the Cluster.listLogs method and send the following request, e.g., via cURL:

   curl \
       --request GET \
       --header "Authorization: Bearer $IAM_TOKEN" \
       --url 'https://mdb.api.cloud.yandex.net/managed-kafka/v1/clusters/<cluster_ID>:logs' \
       --url-query columnFilter=<list_of_data_columns> \
       --url-query fromTime=<time_range_left_boundary> \
       --url-query toTime=<time_range_right_boundary>
   

   Where:

   • columnFilter: List of data columns:

     • hostname: Host name.
     • component: Type of component to log, Example: HTTP-Session.
     • message: Message output by the component.
     • query_id: Request ID.
     • severity: Logging level, e.g., Debug.
     • thread: ID of the thread involved in query handling.

     You can specify only one column in the columnFilter parameter. If you want to filter logs by more than one column, provide a list of the columns in several parameters.

   • fromTime: Left boundary of a time range in RFC-3339 format, Example: 2006-01-02T15:04:05Z.
   • toTime: Right boundary of a time range, the format is the same as for fromTime.

   You can get the cluster ID with a list of clusters in the folder.

3. View the server response to make sure the request was successful.

1. Get an IAM token for API authentication and put it into the environment variable:

   export IAM_TOKEN="<IAM_token>"
   

2. Clone the cloudapi repository:

   cd ~/ && git clone --depth=1 https://github.com/yandex-cloud/cloudapi
   

   Below, we assume the repository contents are stored in the ~/cloudapi/ directory.

3. Use the ClusterService/ListLogs call and send the following request, e.g., via gRPCurl:

   grpcurl \
       -format json \
       -import-path ~/cloudapi/ \
       -import-path ~/cloudapi/third_party/googleapis/ \
       -proto ~/cloudapi/yandex/cloud/mdb/kafka/v1/cluster_service.proto \
       -rpc-header "Authorization: Bearer $IAM_TOKEN" \
       -d '{
               "cluster_id": "<cluster_ID>",
               "column_filter": [<list_of_data_columns>],
               "from_time": "<time_range_left_boundary>" \
               "to_time": "<time_range_right_boundary>"
           }' \
       mdb.api.cloud.yandex.net:443 \
       yandex.cloud.mdb.kafka.v1.ClusterService.ListLogs
   

   Where:

   • service_type: Type of the service to request logs for. The only valid value is CLICKHOUSE.

   • column_filter: List of data columns:

     • hostname: Host name.
     • component: Type of component to log, Example: HTTP-Session.
     • message: Message output by the component.
     • query_id: Request ID.
     • severity: Logging level, e.g., Debug.
     • thread: ID of the thread involved in query handling.

     You can specify more than one column in the column_filter parameter if you want to filter logs by multiple columns.

   • from_time: Left boundary of a time range in RFC-3339 format, Example: 2006-01-02T15:04:05Z.
   • to_time: Right boundary of a time range, the format is the same as for from_time.

   You can get the cluster ID with a list of clusters in the folder.

4. View the server response to make sure the request was successful.

1. Get an IAM token for API authentication and put it into the environment variable:

   export IAM_TOKEN="<IAM_token>"
   

2. Use the Cluster.streamLogs method and send the following request, e.g., via cURL:

   curl \
       --request GET \
       --header "Authorization: Bearer $IAM_TOKEN" \
       --url 'https://mdb.api.cloud.yandex.net/managed-kafka/v1/clusters/<cluster_ID>:stream_logs' \
       --url-query columnFilter=<list_of_data_columns> \
       --url-query fromTime=<time_range_left_boundary> \
       --url-query toTime=<time_range_right_boundary> \
       --url-query filter=<log_filter>
   

   Where:

   • columnFilter: List of data columns:

     • hostname: Host name.
     • component: Type of component to log, Example: HTTP-Session.
     • message: Message output by the component.
     • query_id: Request ID.
     • severity: Logging level, e.g., Debug.
     • thread: ID of the thread involved in query handling.

     You can specify only one column in the columnFilter parameter. If you want to filter logs by more than one column, provide a list of the columns in several parameters.

   • fromTime: Left boundary of a time range in RFC-3339 format, Example: 2006-01-02T15:04:05Z.

   • toTime: Right boundary of a time range, the format is the same as for fromTime.

     If you omit this parameter, new logs will be sent to the log stream as they arrive. Semantically, this behavior is similar to tail -f.

   • filter: Log filter. You can filter logs so that the stream contains only the logs you need.

     For more information about filters and their syntax, see the API reference.

     Tip

     A filter can contain quotation marks and other characters. Escape them if you need to.

     Supported filters:

     • message.hostname: Filtering by host name.
     • message.severity: Filtering by logging level.

   You can get the cluster ID with a list of clusters in the folder.

3. View the server response to make sure the request was successful.

1. Get an IAM token for API authentication and put it into the environment variable:

   export IAM_TOKEN="<IAM_token>"
   

2. Clone the cloudapi repository:

   cd ~/ && git clone --depth=1 https://github.com/yandex-cloud/cloudapi
   

   Below, we assume the repository contents are stored in the ~/cloudapi/ directory.

3. Use the ClusterService/StreamLogs call and send the following request, e.g., via gRPCurl:

   grpcurl \
       -format json \
       -import-path ~/cloudapi/ \
       -import-path ~/cloudapi/third_party/googleapis/ \
       -proto ~/cloudapi/yandex/cloud/mdb/kafka/v1/cluster_service.proto \
       -rpc-header "Authorization: Bearer $IAM_TOKEN" \
       -d '{
               "cluster_id": "<cluster_ID>",
               "column_filter": [<list_of_data_columns>],
               "from_time": "<time_range_left_boundary>",
               "to_time": "<time_range_right_boundary>",
               "filter": "<log_filter>"
           }' \
       mdb.api.cloud.yandex.net:443 \
       yandex.cloud.mdb.kafka.v1.ClusterService.StreamLogs
   

   Where:

   • column_filter: List of data columns:

     • hostname: Host name.
     • component: Type of component to log, Example: HTTP-Session.
     • message: Message output by the component.
     • query_id: Request ID.
     • severity: Logging level, e.g., Debug.
     • thread: ID of the thread involved in query handling.

     You can specify more than one column in the column_filter parameter if you want to filter logs by multiple columns.

   • from_time: Left boundary of a time range in RFC-3339 format, Example: 2006-01-02T15:04:05Z.

   • to_time: Right boundary of a time range, the format is the same as for from_time.

     If you omit this parameter, new logs will be sent to the log stream as they arrive. Semantically, this behavior is similar to tail -f.

   • filter: Log filter. You can filter logs so that the stream contains only the logs you need.

     Tip

     A filter can contain quotation marks and other characters. Escape them if you need to.

     Supported filters:

     • message.hostname: Filtering by host name.
     • message.severity: Filtering by logging level.

     For more information about filters and their syntax, see the API reference.

   You can get the cluster ID with a list of clusters in the folder.

4. View the server response to make sure the request was successful.