Yandex Monitoring metric reference

This section describes Data Transfer metrics exported to Monitoring.

The name label contains the metric name.

All Data Transfer metrics share the following labels:

Label	Value
service	ID of the service: data-transfer.
job_index	Worker index to distinguish workers used for parallel data copying
src_id	Source ID
target_type	Target type, e.g., mongo
resource_id	Transfer or endpoint ID depending on resource type.
resource_type	Type of resource the operation was started or completed on. The possible values are endpoint and transfer.
dst_id	Target ID
source_type	Source type, e.g., mongo
operation_type	Type of operation either started or completed. The values correspond to possible operations on resource_type. For a transfer: Activate, Upload, ReUpload, Start, Restart, Stop, Verify, AddTables, RemoveTables, Deactivate, Checksum, Replication, Termination, TestEndpoint. For an endpoint: Create, Update, Delete.

CPU metricsCPU metrics

CPU core workload.

Labels shared by all CPU metrics:

Label	Value
component	System component, e.g., psutil

Name Type, units	Description
cpu.counts COUNTER, count	Number of CPUs allocated for data transfer in a worker
proc.cpu DGAUGE, %	CPU core workload.
proc.descriptors DGAUGE, count	Number of open file descriptors
proc.ram DGAUGE, bytes	RAM usage

RAM metricsRAM metrics

Labels shared by all RAM metrics:

Label	Value
component	System component, e.g., psutil

Name Type, units	Description
mem.available COUNTER, bytes	RAM usage, available usage type
mem.percentage DGAUGE, %	Percentage of RAM usage
mem.used COUNTER, bytes	RAM usage, used usage type

Service metricsService metrics

Name Type, units	Description Labels
fallbacks.source.deepness DGAUGE, count	Current number of items in the source-side fallback queue
fallbacks.source.errors COUNTER, count	Number of source-side fallback errors
fallbacks.source.items COUNTER, count	Total items added to the source-side fallback queue
fallbacks.target.deepness DGAUGE, count	Current number of items in the target-side fallback queue
fallbacks.target.errors COUNTER, count	Number of target-side fallback errors
fallbacks.target.items COUNTER, count	Total items added to the target-side fallback queue
logger.bytes_written COUNTER, bytes	Total size of written logs
logger.field_truncated_size_hist IGAUGE, bytes	Histogram of truncated log fields. This metric features the bin label containing histogram buckets.
logger.leaked_count COUNTER, count	Number of log leaks
logger.leaked_size COUNTER, bytes	Size of log leaks
logger.leaked_size_hist IGAUGE, bytes	Histogram of log leaks. This metric features the bin label containing histogram buckets.
logger.success_size_hist IGAUGE, bytes	Histogram of written logs. This metric features the bin label containing histogram buckets.
middleware.error_tracker.failures COUNTER, count	Number of errors in the error tracker. This metric features the component label containing a system component, e.g., middleware_filter.
middleware.error_tracker.success COUNTER, count	Number of successful operations in the error tracker This metric features the component label containing a system component, e.g., middleware_filter.
middleware.filter.dropped COUNTER, count	Number of dropped items. This metric features the component label containing a system component, e.g., middleware_filter.
publisher.consumer.active DGAUGE, count	Number of active consumers (data processing threads)
publisher.consumer.compress_ratio DGAUGE, %	Data compression ratio during data transfer
publisher.consumer.ddl_error COUNTER, count	Number of DDL errors
publisher.consumer.error COUNTER, count	Total errors in consumers
publisher.consumer.extracted_bytes DGAUGE, bytes	Size of data extracted from the source
publisher.consumer.fatal COUNTER, count	Number of fatal errors requiring system specialist involvement
publisher.consumer.log_usage_bytes DGAUGE, bytes	Size of buffer or write ahead log (when supported) in the source
publisher.consumer.read_bytes DGAUGE, bytes	Size of read data
publisher.data.bytes COUNTER, bytes	Size of data read from the source
publisher.data.changeitems COUNTER, count	Number of source events generated for a transfer. Apart from the data to transfer, these events may include housekeeping operations.
publisher.data.parsed_rows COUNTER, count	Number of rows successfully parsed and processed
publisher.data.transactions COUNTER, count	Number of data transactions processed during transfer
publisher.data.unparsed_rows COUNTER, count	Number of data rows that could not be successfully parsed
publisher.time.delay_ms DGAUGE, milliseconds	Delay during data transfer
publisher.time.parse_ms DGAUGE, milliseconds	Time spent on data parsing
publisher.time.push_ms DGAUGE, milliseconds	Time spent sending data to target
publisher.time.transform_ms DGAUGE, milliseconds	Time spent on data transformation
replication.running DGAUGE, 0/1	Current replication state. It can take either of these two values: 1: Launched. 0: Stopped.
replication.start.unix DGAUGE, number	Replication start timestamp in Unix epoch format
runtime.alloc COUNTER, bytes	Memory allocated but not yet released. This metric features the component label containing a system component, e.g., psutil.
runtime.heapIdle COUNTER, bytes	Dynamic memory allocation, which is currently idle. This metric features the component label containing a system component, e.g., psutil.
runtime.heapInuse COUNTER, bytes	Dynamic memory allocation, which is actively used. This metric features the component label containing a system component, e.g., psutil.
runtime.numGC COUNTER, count	Number of garbage collection (GC) cycles since measurement initiation. This metric features the component label containing a system component, e.g., psutil.
runtime.sys COUNTER, bytes	Total system memory in use. This metric features the component label containing a system component, e.g., psutil.
runtime.totalAlloc COUNTER, bytes	Total memory allocated over the run time. This metric features the component label containing a system component, e.g., psutil.
sinker.pusher.data.changeitems COUNTER, count	Number of events written to the target. Apart from the data to transfer, these events may include housekeeping operations.
sinker.pusher.data.row_events_pushed COUNTER, count	Number of rows sent to the target
sinker.pusher.time.batch_push_distribution_sec IGAUGE, seconds	Full time it takes to write a batch to the target, including data preprocessing. This metric features the bin label containing histogram buckets.
sinker.pusher.time.row_lag_sec IGAUGE, seconds	Time lag between when the records appear in the target and when they appear in the source. This metric features the bin label containing histogram buckets.
sinker.pusher.time.row_max_lag_sec DGAUGE, seconds	Maximum data lag
sinker.pusher.time.row_max_read_lag_sec DGAUGE, seconds	Maximum lag between when the data appears in the source and when it is read.
sinker.table.deleted_rows COUNTER, count	Number of deleted table rows. This metric features the table label containing a DB table or collection.
sinker.table.error COUNTER, count	Number of errors that occurred while processing the table. This metric features the table label containing a DB table or collection.
sinker.table.rows COUNTER, count	50 tables with the maximum number of rows written to the target. This metric features the table label containing a DB table or collection.
sinker.table.updated_rows COUNTER, count	Number of updated table rows. This metric features the table label containing a DB table or collection.
sinker.table.upserted_rows COUNTER, count	Number of inserted or updated (upsert) table rows. This metric features the table label containing a DB table or collection.
sinker.time.bulkPrepare DGAUGE, seconds	Time to prepare a data batch for writing
sinker.time.bulkWrite DGAUGE, seconds	Time to write a data batch
sinker.time.push DGAUGE, seconds	Total data write time
sinker.transactions.inflight COUNTER, count	Number of active transactions
sinker.transactions.total COUNTER, count	Total completed transactions
storage.diff_perc DGAUGE, %	Percentage difference between the number of source and target records. This metric features the table label containing a DB table or collection.
storage.source_rows DGAUGE, count	Number of data source rows. This metric features the table label containing a DB table or collection.
storage.target_rows DGAUGE, count	Number of data target rows. This metric features the table label containing a DB table or collection.
task.snapshot.remainder.table DGAUGE, count	Number of rows awaiting transfer. This metric features the table label containing a DB table or collection.
task.snapshot.reminder.total DGAUGE, count	Total remaining rows to transfer. This metric features the table label containing a DB table or collection.
task.status DGAUGE, 0/1	Status of the operation in progress. The possible values include: 1: Active issue. 0: Inactive issue.

Useful linksUseful links

Transfer state monitoring