Back to Blog
By AriesZhou · · 4 min read

Understanding Load Average

System

Load average includes not only processes currently using the CPU, but also processes waiting for the CPU and waiting for I/O.

What is load average

When we use the top or uptime command to check system load, we usually pay attention to the load average parameter. But do you really understand what the three numbers after this parameter mean? Quite a few people simply interpret load average as CPU usage per unit of time, but that is not the case.

root@zwx-virtual-machine:/# uptime
 11:06:40 up 1 day, 12 min,  1 user,  load average: 0.09, 0.04, 0.01

The three numbers after load average in the figure are the average load over the past 1 minute, 5 minutes, and 15 minutes, respectively. Simply put, it represents the average number of processes per unit of time that are in the runnable state and the uninterruptible state, that is, the average number of active processes, and it has no direct relationship with CPU usage.

Runnable processes are those currently using the CPU or waiting for the CPU, in other words, processes in the R/Running/Runnable state as shown by the ps command.

Uninterruptible processes are those in critical kernel-mode paths that cannot be interrupted, such as waiting for I/O responses from hardware devices, the D-state processes we see in the ps command.

How high is too high for load average?

Load average is the average number of active processes, and since it averages active processes, the question is: average relative to what? Obviously, the CPU. So evaluating load average requires considering the number of CPUs. When the load average equals the number of CPUs, that is the ideal load state. For example, a load average of 4 means that on a 4-CPU system, all CPUs are exactly fully occupied; on an 8-CPU system, 50% of the CPUs are idle; on a 2-CPU system, half of the processes failed to compete for CPU.

At this point, we can make a preliminary judgment: when the load average is greater than the number of CPUs, the system has already experienced overload. But that alone is not enough. Load average has three values, and looking at only one is clearly insufficient. Just as evaluating your meals for a day requires considering breakfast, lunch, and dinner together to be objective, evaluating load average also requires considering all three values together.

  • These three values represent the load average over 1 minute, 5 minutes, and 15 minutes. If the three are roughly the same, the system load is relatively stable.
  • If the 1-minute value is much lower than the 15-minute value, the system had a significant load over the past 15 minutes but the trend is downward.
  • If the 1-minute value is much higher than the 15-minute value, the system load has been increasing in the last minute. This increase may be temporary or sustained, so continuous observation is needed in this case.

An example

On a single-CPU system, uptime shows load average: 0.53, 1.25, 5.47. This means that in the past 1 minute, the system was 47% idle; in the past 5 minutes, there was 25% overload; and in the past 15 minutes, there was 447% overload. Overall, the system load average is decreasing.

We all know that excessive load causes processes to respond more slowly, which in turn affects normal service functionality. So at what point should we take effective measures to reduce the load average? Based on the experience of seasoned engineers, when the load average exceeds 70% of the number of CPUs, you should begin analyzing and troubleshooting the high-load problem.

Summary

Load average provides a quick way to check overall system performance and reflects the overall load situation. But looking at load average alone cannot directly reveal the source of a bottleneck. Therefore, when interpreting load average, keep the following points in mind:

  • A high load average may be caused by CPU-intensive processes.
  • High load average does not necessarily mean high CPU usage; it can also indicate I/O contention.
  • When you see a high load average, you can use tools like mpstat and pidstat to help identify the source of the load.