Amazon Elastic Compute Cloud (Amazon EC2) accelerated computing instances run some of the most demanding workloads customers operate, from large model training and distributed fine-tuning to high-throughput, live inference. These customers seek high availability and performance to make sure their training and inference run efficiently. To meet their performance goals, customers require visibility into the efficiency and performance of each accelerated computing chip in their fleet.
For example, central infrastructure or machine learning teams often operate accelerated computing capacity for many application teams. These infrastructure teams need a clear view of utilization and health across every instance in their fleet even if they don’t develop the applications running on those instances. They are often responsible for how efficiently that capacity is used and for hardware health, while application teams own the workload. Customers asked us to provide foundational telemetry for their accelerated computing efficiency and status that doesn’t require setup within the instance and complements their in-guest tooling.
Amazon EC2 now reports one-minute graphics processing unit (GPU) metrics through Detailed Monitoring and surfaces recoverable conditions with performance status checks. One-minute GPU metrics in Amazon EC2 Detailed Monitoring report per-GPU utilization and thermal data to Amazon CloudWatch, so you can identify underused capacity and tune workloads for cost and throughput. Performance status checks report recoverable hardware conditions, such as a GPU memory error, to help you manage GPU performance.
Feature overviews
Amazon EC2 reports GPU metrics and performance status checks without an in-guest agent and without using instance compute. These capabilities are available on supported NVIDIA GPU-based accelerated computing instance types. For the metrics supported on each instance family, see the Amazon EC2 Detailed Monitoring documentation.
One-minute accelerated computing metrics extend EC2 Detailed Monitoring and are published to CloudWatch as OpenTelemetry Metrics or CloudWatch Metrics. The metric name you add to a category determines which one CloudWatch publishes: include GPUUtilization for a CloudWatch Metric, instance.gpu.utilization for an OpenTelemetry Metric, or both names to publish both. Amazon EC2 publishes each metric to CloudWatch at two levels: one series for the instance, identified by InstanceId, and one series for each GPU, identified by InstanceId and GpuId. GpuId is the GPU UUID, which you can match to a device by running nvidia-smi -L on the instance.
Performance status checks add a fifth status check to the existing Amazon EC2 product check suite of system, instance, attached EBS, and application status checks. The check reports recoverable hardware issues that affect accelerated computing performance while the instance might remain running and reachable. It surfaces through the EC2 console, through DescribeInstanceStatus, and in CloudWatch metrics.
Evaluating accelerated computing utilization and performance with EC2 Detailed Monitoring
Amazon EC2 Detailed Monitoring reports per-GPU measurements, including compute activity, memory use, temperature, and data movement.
GPU utilization metrics report different aspects of GPU use. GPUPowerUtilization is available by default and reports active power use. Power utilization scales with the depth of work the GPU performs. Compare it with more granular utilization metrics to see how efficiently your workload uses the GPUs. GPU utilization shows the percentage of time the GPU was running work. Because a busy GPU might use only a fraction of its streaming multiprocessors (SMs), compare it with SM utilization to see how much of the compute capacity the workload uses. Memory utilization shows how much of the GPU’s memory capacity is in use, helping you assess memory headroom. Context accumulated time is a running total of the time a process has held the GPU open, whether or not it ran work. Use the CloudWatch metric math expression RATE(m1)/1000000, where m1 is GPUContextAccumulatedTime, to compare context activity with GPU utilization. A result close to 1 with low GPU utilization indicates that a GPU context is held open but idle.
GPU temperature is reported as a percentage of the GPU’s thermal throttle limit, and memory temperature as a percentage of the memory’s critical temperature. Both metrics can report values above 100 percent when a GPU or its memory runs above the limit. A GPU reduces its clocks when it reaches its throttle limit, so a value at or above 100 percent can explain a drop in throughput. You can correlate these measurements with GPU and interconnect utilization metrics to understand workload performance.
Interconnect utilization metrics show how much of the available bandwidth a workload uses for data movement. Each value reports the busier direction, either send or receive. NVLink utilization measures GPU-to-GPU traffic, such as collective operations in distributed training. It reports all of the GPU’s links together. PCIe utilization measures traffic between the host and the GPU.
Getting started with EC2 Detailed Monitoring
Amazon EC2 Detailed Monitoring publishes instance metrics to Amazon CloudWatch at one-minute intervals without running an agent in the guest operating system. You can publish these as OpenTelemetry Metrics or CloudWatch Metrics.
To get started, create a metric category, an Amazon EC2 resource containing the metrics you want to collect. Next, associate the category with one or more instances. For an existing instance, provide the category ID when you enable Detailed Monitoring with the MonitorInstances API. For a new instance, provide the category ID in the Monitoring object when you call RunInstances or configure a launch template.
The following example creates a metric category for GPU utilization, SM utilization, and memory utilization.
The response includes a metric category ID. To associate the category with an existing instance, use MonitorInstances.
To associate the category when you launch an instance, include the category ID in the Monitoring object.
To apply a category to every instance launched from a launch template, include the category ID in the Monitoring object in the launch template data.
Categories apply to many instances at once, unlike per-instance metric settings. Define the metric set once and associate it wherever it applies. Every associated instance then reports the same set of metrics. Changing a category updates the metric set for every associated instance. The change is eventually consistent across running instances. Add the category to a launch template to apply the same metrics to new instances. This gives you one metric definition per workload type instead of one configuration per host. Metrics apply under standard Amazon CloudWatch pricing.
Managing accelerated computing availability with performance status checks
Accelerated computing hardware can experience non-terminal issues that affect performance and can be resolved with a GPU reset or an instance reboot. For example, when a GPU encounters an uncorrectable memory error, it schedules a repair that remaps the affected row to a spare. The repair takes effect the next time the GPU is reset or the instance is rebooted.
Amazon EC2 performance status checks surface these recoverable hardware conditions. For example, when Amazon EC2 detects a pending GPU memory row remap, the instance’s performance status check reports an impaired status and identifies reboot as the recommended remediation.
You can view performance status checks on the Amazon EC2 console or through DescribeInstanceStatus. Amazon EC2 also publishes the StatusCheckFailed_Performance metric to CloudWatch. A failed performance status check also increments the combined StatusCheckFailed metric. The following example uses the EC2 DescribeInstanceStatus API:
The following response shows the system, instance, attached EBS, and configured application status checks passing while the performance status check is impaired:
Amazon EC2 sends a scheduled event for the instance if the condition requires further action after a reboot.
To act on these conditions across your accelerated computing instances, create a CloudWatch alarm on StatusCheckFailed_Performance.
- The alarm on
StatusCheckFailed_Performancefires for an instance in a training cluster. - You confirm the condition and the recommended action with
DescribeInstanceStatus, which returns a status of impaired and a remediation of reboot. - You checkpoint the job if applicable, then drain the node and reboot the instance at a planned point. On Amazon Elastic Kubernetes Service (Amazon EKS), cordon and drain the node. In Slurm, set the node to
DRAIN. - After the reboot completes, the performance status check returns to ok.
The following command creates that alarm for a single instance.
A CloudWatch Metrics Insights alarm evaluates a single query across all matching instances, so one alarm can cover a large fleet. To drive automation without an alarm, poll the DescribeInstanceStatus API.
Rebooting the instance is the recommended remediation. Alternatively, you can reset the affected GPU while keeping the other GPUs running. On the affected instance, run nvidia-smi -q -d ROW_REMAPPER and find the GPU that reports Pending: Yes. Stop all processes using that GPU, then use root permissions to run nvidia-smi -r -i <GPU_UUID>. The performance status check returns to ok shortly after the reset completes.
Bringing it all together
Accelerated computing metrics and performance status checks address complementary questions. Detailed Monitoring metrics help you find idle or underused capacity and tune workloads for cost and throughput. Performance status checks surface recoverable conditions and recommend remediation. Together, these EC2 signals report utilization and status across the capacity infrastructure teams manage, while in-guest tooling provides workload-specific analysis.
For example, a central infrastructure team monitors how efficiently workloads use the accelerated computing instances it manages for several application teams. It finds that one team’s GPUs report lower GPU utilization and power utilization than comparable workloads run by other teams. The infrastructure team shares the finding and relevant workload-tuning practices with the application team. After the application team updates that workload, the infrastructure team uses the same CloudWatch metrics to evaluate the change. The infrastructure team also monitors StatusCheckFailed_Performance across the fleet. When a check reports an impaired status and recommends a reboot, the orchestrator saves the active job, stops placing new work on the instance, and reboots it. After the check returns to ok, the orchestrator makes the instance available for new jobs. Together, these workflows help the team improve GPU utilization, protect job progress, and return affected capacity to service.
Amazon EC2 Detailed Monitoring reports accelerated computing metrics once per minute at the instance and GPU levels. Performance status checks report recoverable conditions at the instance level. Infrastructure teams can use these EC2 signals to monitor utilization and status across the capacity they manage. In-guest tools add process, container, Kubernetes pod, and sub-minute detail.
Amazon EC2 Detailed Monitoring reports metrics for your instances, including GPU utilization metrics reported per GPU. Performance status checks tell you when accelerated computing hardware needs attention. These capabilities help you evaluate utilization and manage the efficiency of your accelerated computing fleets. To learn more, see the Amazon EC2 Detailed Monitoring documentation and the EC2 status checks documentation.


