Yandex Data Processing metrics
Written by
Updated at July 8, 2024
This section describes the Yandex Data Processing metrics delivered to Monitoring.
The name of the metric is written in the name
label.
Common labels for all Yandex Data Processing metrics:
Label | Value |
---|---|
service | Service ID: data-proc |
resource_type | Resource type: cluster |
resource_id | Cluster ID |
zone_id | Placement zone |
host | Host FQDN |
HDFS metrics
Name Type, units |
Description |
---|---|
dfs.cluster.Free_bytes DGAUGE , bytes |
Space available in HDFS |
dfs.cluster.NonDfsUsedSpace_bytes DGAUGE , bytes |
Space used by data storage subclusters (DataNode), not available for HDFS |
dfs.cluster.PercentRemaining DGAUGE , % |
Space available in HDFS |
dfs.cluster.PercentUsed DGAUGE , % |
Space used in HDFS |
dfs.cluster.Total_bytes DGAUGE , bytes |
HDFS size |
dfs.cluster.Used_bytes DGAUGE , bytes |
Space used in HDFS |
Disk metrics
Name Type, units |
Description |
---|---|
system.disk.free_bytes DGAUGE , bytes |
Space available in system storage |
system.disk.inodes_free DGAUGE , number |
Number of free index descriptors |
system.disk.inodes_total DGAUGE , number |
Total number of index descriptors |
system.disk.inodes_used DGAUGE , number |
Number of used index descriptors |
system.disk.inodes_used_percent DGAUGE , % |
Percentage of used index descriptors |
system.disk.total_bytes DGAUGE , bytes |
System storage size |
system.disk.used_bytes DGAUGE , bytes |
Used disk space |
system.disk.used_percent DGAUGE , % |
Used disk space |
YARN metrics
Name Type, units |
Description |
---|---|
yarn.cluster.activeNodes DGAUGE , number |
Number of active nodes |
yarn.cluster.allocatedMB DGAUGE , MB |
Allocated memory |
yarn.cluster.allocatedVirtualCores DGAUGE , number |
Number of allocated virtual cores |
yarn.cluster.appsCompleted DGAUGE , number |
Apps completed successfully |
yarn.cluster.appsFailed DGAUGE , number |
Apps failed |
yarn.cluster.appsKilled DGAUGE , number |
Apps killed |
yarn.cluster.appsPending DGAUGE , number |
Apps enqueued |
yarn.cluster.appsRunning DGAUGE , number |
Apps running |
yarn.cluster.appsSubmitted DGAUGE , number |
Apps started |
yarn.cluster.availableMB DGAUGE , MB |
Available memory |
yarn.cluster.availableVirtualCores DGAUGE , number |
Number of available virtual cores |
yarn.cluster.containersAllocated DGAUGE , number |
Number of allocated containers |
yarn.cluster.containersPending DGAUGE , number |
Containers enqueued |
yarn.cluster.containersReserved DGAUGE , number |
Containers reserved |
yarn.cluster.decommissionedNodes DGAUGE , number |
Nodes decommissioned |
yarn.cluster.decommissioningNodes DGAUGE , number |
Nodes under decommissioning |
yarn.cluster.lostNodes DGAUGE , number |
Nodes lost |
yarn.cluster.rebootedNodes DGAUGE , number |
Nodes rebooted |
yarn.cluster.reservedMB DGAUGE , MB |
Reserved memory |
yarn.cluster.reservedVirtualCores DGAUGE , number |
Number of reserved virtual cores |
yarn.cluster.shutdownNodes DGAUGE , number |
Nodes shut down |
yarn.cluster.totalAllocatedContainersAcrossPartition DGAUGE , number |
Containers allocated across partitions |
yarn.cluster.totalMB DGAUGE , MB |
Total memory |
yarn.cluster.totalNodes DGAUGE , number |
Total number of nodes |
yarn.cluster.totalReservedResourcesAcrossPartition_memory DGAUGE |
Memory reserved across partitions |
yarn.cluster.totalReservedResourcesAcrossPartition_resourceInformations_resourceInformation_0_maximumAllocation DGAUGE |
Maximum amount of type 0 resources reserved in all partitions |
yarn.cluster.totalReservedResourcesAcrossPartition_resourceInformations_resourceInformation_0_minimumAllocation DGAUGE |
Minimum amount of type 0 resources reserved in all partitions |
yarn.cluster.totalReservedResourcesAcrossPartition_resourceInformations_resourceInformation_0_value DGAUGE |
Current amount of type 0 resources reserved in all partitions |
yarn.cluster.totalReservedResourcesAcrossPartition_resourceInformations_resourceInformation_1_maximumAllocation DGAUGE |
Maximum amount of type 1 resources reserved in all partitions |
yarn.cluster.totalReservedResourcesAcrossPartition_resourceInformations_resourceInformation_1_minimumAllocation DGAUGE |
Minimum amount of type 1 resources reserved in all partitions |
yarn.cluster.totalReservedResourcesAcrossPartition_resourceInformations_resourceInformation_1_value DGAUGE |
Current amount of type 1 resources reserved in all partitions |
yarn.cluster.totalReservedResourcesAcrossPartition_vCores DGAUGE , number |
Virtual cores reserved across partitions |
yarn.cluster.totalVirtualCores DGAUGE , number |
Total number of virtual cores |
yarn.cluster.unhealthyNodes DGAUGE , number |
Unhealthy nodes |
yarn.cluster.utilizedMBPercent DGAUGE , % |
Utilized memory |
yarn.cluster.utilizedVirtualCoresPercent DGAUGE , % |
Utilized virtual cores |
Other metrics
Name Type, units |
Description |
---|---|
dataproc.cluster.health_status IGAUGE , 0/1/2 |
Cluster health and technical condition:0 : Cluster is out of order (all its hosts are down).1 : Cluster is not running at full capacity (at least one of its hosts is other than ALIVE ).2 : Cluster is running normally. |
dataproc.cluster.neededAutoscalingNodesNumber DGAUGE , number |
Yandex Data Processing service metric for scaling by default |