Yandex Data Processing metrics
Written by
Updated at December 16, 2024
This section describes Yandex Data Processing metrics delivered to Monitoring.
The name of the metric is written in the name label.
Common labels for all Yandex Data Processing metrics:
| Label | Value |
|---|---|
| service | Service ID: data-proc |
| resource_type | Resource type: cluster |
| resource_id | Cluster ID |
| zone_id | Placement zone |
| host | Host FQDN |
HDFS metrics
| Name Type, units |
Description |
|---|---|
dfs.cluster.Free_bytesDGAUGE, bytes |
Space available in HDFS |
dfs.cluster.NonDfsUsedSpace_bytesDGAUGE, bytes |
Space used by data storage subclusters (DataNode), not available for HDFS |
dfs.cluster.PercentRemainingDGAUGE, % |
Space available in HDFS |
dfs.cluster.PercentUsedDGAUGE, % |
Space used in HDFS |
dfs.cluster.Total_bytesDGAUGE, bytes |
HDFS size |
dfs.cluster.Used_bytesDGAUGE, bytes |
Space used in HDFS |
Disk metrics
| Name Type, units |
Description |
|---|---|
system.disk.free_bytesDGAUGE, bytes |
Space available in system storage |
system.disk.inodes_freeDGAUGE, number |
Number of free index descriptors |
system.disk.inodes_totalDGAUGE, number |
Total number of index descriptors |
system.disk.inodes_usedDGAUGE, number |
Number of used index descriptors |
system.disk.inodes_used_percentDGAUGE, % |
Percentage of used index descriptors |
system.disk.total_bytesDGAUGE, bytes |
System storage size |
system.disk.used_bytesDGAUGE, bytes |
Used disk space |
system.disk.used_percentDGAUGE, % |
Used disk space |
YARN metrics
| Name Type, units |
Description |
|---|---|
yarn.cluster.activeNodesDGAUGE, number |
Number of active nodes |
yarn.cluster.allocatedMBDGAUGE, MB |
Allocated memory |
yarn.cluster.allocatedVirtualCoresDGAUGE, number |
Number of allocated virtual cores |
yarn.cluster.appsCompletedDGAUGE, number |
Apps completed successfully |
yarn.cluster.appsFailedDGAUGE, number |
Apps failed |
yarn.cluster.appsKilledDGAUGE, number |
Apps killed |
yarn.cluster.appsPendingDGAUGE, number |
Apps enqueued |
yarn.cluster.appsRunningDGAUGE, number |
Apps running |
yarn.cluster.appsSubmittedDGAUGE, number |
Apps started |
yarn.cluster.availableMBDGAUGE, MB |
Available memory |
yarn.cluster.availableVirtualCoresDGAUGE, number |
Number of available virtual cores |
yarn.cluster.containersAllocatedDGAUGE, number |
Number of allocated containers |
yarn.cluster.containersPendingDGAUGE, number |
Containers enqueued |
yarn.cluster.containersReservedDGAUGE, number |
Containers reserved |
yarn.cluster.decommissionedNodesDGAUGE, number |
Nodes decommissioned |
yarn.cluster.decommissioningNodesDGAUGE, number |
Nodes under decommissioning |
yarn.cluster.lostNodesDGAUGE, number |
Nodes lost |
yarn.cluster.rebootedNodesDGAUGE, number |
Nodes rebooted |
yarn.cluster.reservedMBDGAUGE, MB |
Reserved memory |
yarn.cluster.reservedVirtualCoresDGAUGE, number |
Number of reserved virtual cores |
yarn.cluster.shutdownNodesDGAUGE, number |
Nodes shut down |
yarn.cluster.totalAllocatedContainersAcrossPartitionDGAUGE, number |
Containers allocated across partitions |
yarn.cluster.totalMBDGAUGE, MB |
Total memory |
yarn.cluster.totalNodesDGAUGE, number |
Total number of nodes |
yarn.cluster.totalReservedResourcesAcrossPartition_memoryDGAUGE |
Memory reserved across partitions |
yarn.cluster.totalReservedResourcesAcrossPartition_resourceInformations_resourceInformation_0_maximumAllocationDGAUGE |
Maximum amount of type 0 resources reserved in all partitions |
yarn.cluster.totalReservedResourcesAcrossPartition_resourceInformations_resourceInformation_0_minimumAllocationDGAUGE |
Minimum amount of type 0 resources reserved in all partitions |
yarn.cluster.totalReservedResourcesAcrossPartition_resourceInformations_resourceInformation_0_valueDGAUGE |
Current amount of type 0 resources reserved in all partitions |
yarn.cluster.totalReservedResourcesAcrossPartition_resourceInformations_resourceInformation_1_maximumAllocationDGAUGE |
Maximum amount of type 1 resources reserved in all partitions |
yarn.cluster.totalReservedResourcesAcrossPartition_resourceInformations_resourceInformation_1_minimumAllocationDGAUGE |
Minimum amount of type 1 resources reserved in all partitions |
yarn.cluster.totalReservedResourcesAcrossPartition_resourceInformations_resourceInformation_1_valueDGAUGE |
Current amount of type 1 resources reserved in all partitions |
yarn.cluster.totalReservedResourcesAcrossPartition_vCoresDGAUGE, number |
Virtual cores reserved across partitions |
yarn.cluster.totalVirtualCoresDGAUGE, number |
Total number of virtual cores |
yarn.cluster.unhealthyNodesDGAUGE, number |
Unhealthy nodes |
yarn.cluster.utilizedMBPercentDGAUGE, % |
Utilized memory |
yarn.cluster.utilizedVirtualCoresPercentDGAUGE, % |
Utilized virtual cores |
Other metrics
| Name Type, units |
Description |
|---|---|
dataproc.cluster.health_statusIGAUGE, 0/1/2 |
Cluster health and technical condition:0: Cluster is out of order (all its hosts are down).1: Cluster is not running at full capacity (at least one of its hosts is other than ALIVE).2: Cluster is running normally. |
dataproc.cluster.neededAutoscalingNodesNumberDGAUGE, number |
Yandex Data Processing service metric for scaling by default |