Contact UsGet started

Yandex Data Transfer metrics

Written by
Updated at August 21, 2024

This section describes the Data Transfer metrics delivered to Monitoring.

The metric name is written to the name label.

All Data Transfer metrics have common labels:

Label Value
service Service ID: data-transfer
job_index Worker index to distinguish workers used for parallel data copying.
src_id Source ID
target_type Target type, e.g., mongo
resource_id Data Transfer transfer ID
dst_id Target ID
source_type Source type, e.g., mongo
operation_type (except for the replication.* metrics) Operation type, e.g., Activate

CPU metrics

Processor core workload

Common labels for all CPU metrics:

Label Value
component System component, e.g., psutil
Name
Type, units		 Description
cpu.counts
COUNTER, number		 Number of CPUs allocated for data transfer in the worker
proc.cpu
DGAUGE, %		 Processor core workload
proc.descriptors
DGAUGE, number		 Number of open file descriptors
proc.ram
DGAUGE, bytes		 RAM usage

RAM metrics

Common labels for all RAM metrics:

Label Value
component System component, e.g., psutil
Name
Type, units		 Description
mem.available
COUNTER, bytes		 RAM usage, available usage type
mem.percentage
DGAUGE, %		 Percentage of RAM usage
mem.used
COUNTER, bytes		 RAM usage, used usage type

Service metrics

Name
Type, units		 Description
Labels
fallbacks.source.deepness
DGAUGE, number		 Current number of elements in the fallback queue on the source side
fallbacks.source.errors
COUNTER, number		 Number of errors on the source side during the fallback process
fallbacks.source.items
COUNTER, number		 Total number of elements added to the fallback queue on the source side
fallbacks.target.deepness
DGAUGE, number		 Current number of elements in the fallback queue on the target side
fallbacks.target.errors
COUNTER, number		 Number of errors on the target side during the fallback process
fallbacks.target.items
COUNTER, number		 Total number of elements added to the fallback queue on the target side
logger.bytes_written
COUNTER, bytes		 Total volume of recorded logs
logger.field_truncated_size_hist
IGAUGE, bytes		 Histogram of truncated fields in logs
bin label: Histogram buckets
logger.leaked_count
COUNTER, bytes		 Number of leaks in logs
logger.leaked_size
COUNTER, bytes		 Volume of leaks in logs
logger.leaked_size_hist
IGAUGE, bytes		 Histogram of leaks in logs
bin label: Histogram buckets
logger.success_size_hist
IGAUGE, bytes		 Histogram of recorded logs
bin label: Histogram buckets
middleware.error_tracker.failures
COUNTER, bytes		 Number of errors in the error tracker
component label: System component, e.g., middleware_filter
middleware.error_tracker.success
COUNTER, number		 Number of successful operations in the error tracker
component label: System component, e.g., middleware_filter
middleware.filter.dropped
COUNTER, number		 Number of discarded elements
component label: System component, e.g., middleware_filter
publisher.consumer.active
DGAUGE, number		 Number of active consumers (data processing threads)
publisher.consumer.compress_ratio
DGAUGE, %		 Data compression ratio during data transfer
publisher.consumer.ddl_error
COUNTER, number		 Number of errors related to DDL (Data Definition Language) operations
publisher.consumer.error
COUNTER, number		 Total number of errors encountered by consumers
publisher.consumer.extracted_bytes
DGAUGE, bytes		 Amount of data extracted from the source
publisher.consumer.fatal
COUNTER, number		 Number of fatal errors requiring agent involvement
publisher.consumer.log_usage_bytes
DGAUGE, bytes		 Size of buffer or write ahead log (when supported) in the source
publisher.consumer.read_bytes
DGAUGE, bytes		 Amount of read data
publisher.data.bytes
COUNTER, bytes		 Amount of data read from the source
publisher.data.changeitems
COUNTER, number		 Number of source events generated for a transfer (apart from the data to transfer, these events may include housekeeping operations)
publisher.data.parsed_rows
COUNTER, number		 Number of rows successfully processed after they were parsed
publisher.data.transactions
COUNTER, number		 Number of data transactions processed during transfer
publisher.data.unparsed_rows
COUNTER, number		 Number of data rows that could not be successfully parsed
publisher.time.delay_ms
DGAUGE, milliseconds		 Delay during data transfer
publisher.time.parse_ms
DGAUGE, milliseconds		 Time spent on data parsing
publisher.time.push_ms
DGAUGE, milliseconds		 Time spent sending data to target
publisher.time.transform_ms
DGAUGE, milliseconds		 Time spent on data transformation
replication.running
DGAUGE, 0/1		 Current replication state
Takes on these values:
  • 1: Launched
  • 0: Stopped
replication.start.unix
DGAUGE, number		 Replication start time label in Unix epoch format
runtime.alloc
COUNTER, bytes		 Total amount of memory allocated but not freed up yet
component label: System component, e.g., psutil
runtime.heapIdle
COUNTER, bytes		 Amount of memory allocated for dynamic memory but not currently in use
component label: System component, e.g., psutil
runtime.heapInuse
COUNTER, bytes		 Amount of memory actively used as dynamic memory
component label: System component, e.g., psutil
runtime.numGC
COUNTER, bytes		 Number of garbage collection (GC) cycles performed since the start of measurement time
component label: System component, e.g., psutil
runtime.sys
COUNTER, bytes		 Total amount of system memory in use
component label: System component, e.g., psutil
runtime.totalAlloc
COUNTER, bytes		 Total amount of memory allocated for the entire run time
component label: System component, e.g., psutil
sinker.pusher.data.changeitems
COUNTER, number		 Number of events written to the target (apart from the data to transfer, these events may include housekeeping operations)
sinker.pusher.data.row_events_pushed
COUNTER, number		 Number of rows sent to target
sinker.pusher.time.batch_push_distribution_sec
IGAUGE, seconds		 Full time it takes to write a batch to the target, including data preprocessing
bin label: Histogram buckets
sinker.pusher.time.row_lag_sec
IGAUGE, seconds		 Time lag between records appearing at target and source
bin label: Histogram buckets
sinker.pusher.time.row_max_lag_sec
DGAUGE, seconds		 Maximum data lag
sinker.pusher.time.row_max_read_lag_sec
DGAUGE, seconds		 Maximum lag between data appearing at source till the read time
sinker.table.deleted_rows
COUNTER, number		 Number of deleted table rows
table label: DB table or collection
sinker.table.rows
COUNTER, number		 50 tables with the maximum number of rows written to the target
table label: DB table or collection
sinker.table.updated_rows
COUNTER, number		 Number of updated table rows
table label: DB table or collection
sinker.table.upserted_rows
COUNTER, number		 Number of table rows inserted or updated (upsert)
table label: DB table or collection
sinker.time.bulkPrepare
DGAUGE, seconds		 Time to prepare a data batch for writing
sinker.time.bulkWrite
DGAUGE, seconds		 Time to write a data batch
sinker.time.push
DGAUGE, seconds		 Total data write operation time
sinker.transactions.inflight
COUNTER, number		 Number of active transactions
sinker.transactions.total
COUNTER, number		 Total number of completed transactions
storage.diff_perc
DGAUGE, %		 Percentage difference between the number of source and target records
table label: DB table or collection
storage.source_rows
DGAUGE, number		 Number of data source rows
table label: DB table or collection
storage.target_rows
DGAUGE, number		 Number of data target rows
table label: DB table or collection
task.snapshot.remainder.table
DGAUGE, number		 Number of rows awaiting transfer
table label: DB table or collection
task.snapshot.reminder.total
DGAUGE, number		 Total number of rows still to be transferred
table label: DB table or collection
task.status
DGAUGE, 0/1		 Status of operation in progress
Takes on these values:
  • 1: Task active
  • 0: Task inactive
Previous
Yandex Data Processing
Next
Cloud DNS