Viewing Apache Kafka® cluster logs
Managed Service for Apache Kafka® allows you to get a cluster log snippet for the selected period and view logs in real time.
Note
Here, the log is the system log of the cluster and its hosts. This log is not related to the partition log for the Apache Kafka® topic where the broker writes messages received from message producers.
Note
Cluster logs are kept for 30 days.
Getting a cluster log
- In the management console, go to the relevant folder.
- In the list of services, select Managed Service for Kafka.
- Click the name of the cluster you need and select the Logs tab.
- Select Origin, Hosts, and ** Severity**.
- Specify the time period for which you want to display the log.
If you do not have the Yandex Cloud command line interface yet, install and initialize it.
The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.
-
View a description of the CLI command to view cluster logs:
yc managed-kafka cluster list-logs --help -
Run the following command to get cluster logs (our example does not contain a complete list of available parameters):
yc managed-kafka cluster list-logs <cluster_name_or_ID> \ --limit <entry_number_limit> \ --columns <log_columns_list> \ --filter <entry_filtration_settings> \ --since <time_range_left_boundary> \ --until <time_range_right_boundary>Where:
-
--limit: limits on the number of entries to output. -
--columns: List of log columns to draw data from.hostname: Host name.message: Message output by the component.severity: Logging level. Output example:INFO.origin: Message origin. Output examples:kafka_serverorkafka_controller.
-
--filter: record filter settings, for example,message.hostname='node1.mdb.yandexcloud.net'. -
--since: Left boundary of a time range in RFC-3339,HH:MM:SSformat or a time interval relative to the current time. Examples:2006-01-02T15:04:05Z,15:04:05,2h,3h30m ago. -
--until: right boundary of a time range, the format is similar to that of--since.
-
You can request the cluster name and ID with a list of clusters in the folder.
To get a cluster log, use the listLogs REST API method for the Cluster resource or the ClusterService/ListLogs gRPC API call and provide the cluster ID in the clusterId request parameter.
You'll get the full cluster log. The number of log entries that the cluster can return must not exceed 100,000 (100 pages of 1000 entries each).
If the log size exceeds this value or you want to get logs for a specific period of time, pass in the fromTime and toTime request parameters the timeframes in RFC-3339 format.
You can get the cluster ID with a list of clusters in the folder.
Getting a log entry stream
This method allows you to get cluster logs in real time.
If you do not have the Yandex Cloud command line interface yet, install and initialize it.
The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.
To view cluster logs as they become available, run this command:
yc managed-kafka cluster list-logs <cluster_name_or_ID> --follow
You can request the cluster name and ID with a list of clusters in the folder.
To get a cluster's log stream, use the streamLogs REST API method for the Cluster resource or the ClusterService/StreamLogs gRPC API call and provide the cluster ID in the clusterId request parameter.
You'll get the full cluster log. The number of log entries that the cluster can return must not exceed 100,000 (100 pages of 1000 entries each).
If the log size exceeds this value or you want to get logs for a specific period of time, pass in the fromTime and toTime request parameters the timeframes in RFC-3339 format.
If you do not set the toTime parameter value, the stream will be receiving new log entries as they appear.
You can get the cluster ID with a list of clusters in the folder.