Understanding top and load average

I’m observing a high load average on a certain machine (about 9) in all three load fields. I understand load as the number of processes in state “run” / currently desiring CPU time. Am I correct at reasoning that if N processes are running on my machine this cannot produce a load greater than N?

Also, does the load count in respect to processes or threads? In other words, can a multithreaded process produce a load greater than 1?

Contents hide

Answers:

Method 1

Method 2

Method 3

Method 4

Method 5

Answers:

Thank you for visiting the Q&A section on Magenaut. Please note that all the answers may not help you solve the issue immediately. So please treat them as advisements. If you found the post helpful (or not), leave a comment & I’ll get back to you as soon as possible.

Method 1

Load average is usually described as “average length of run queue”. So few CPU-consuming processes or threads can raise LA above 1. There is no problem if LA is less than total number of CPU cores. But if it gets higher than number of CPUs, this means some threads/processes will stay in queue, ready to run, but waiting for free CPU.

Method 2

The numbers that are used to calculate load average are tasks in the run or uninterruptable state and the amount of work done in the time slice of the moving average. These tasks can be part of a multithreaded process. The fields are fuzzy the farther back in time due to smoothing results from the algorithm used.

A load of 1 is equal to 100% of one CPUs worth of work. If you had a multithreaded application that managed to have a number of active threads in excess of the number of available CPUs, then you could have a single process drive the load above 1. This would likely be a short term spike and not reflected in the longer time slice views of the load average.

Also, since the load average was developed before there were multi-core systems, its important to divide the load numbers by the total available number of cores. If this is a sustained load of 9 on a 4 socket quad core system, then this is a 9 of 16 load and not really a problem.

Method 3

See kernel/sched/loadavg.c which has a long and excellent comment at the start explaining the derivation of load average from a exponentially decaying average of the number of runnable threads (the “run queue”) plus the number of uninterruptable threads (waiting on I/O or waiting on a lock).

Here’s the essence of the comment, but it is worthwhile reading in full:

 * The global load average is an exponentially decaying average of
 * nr_running + nr_uninterruptible.
 *
 * Once every LOAD_FREQ:
 *     nr_active = 0;
 *     for_each_possible_cpu(cpu)
 *         nr_active += cpu_of(cpu)->nr_running +
 *                      cpu_of(cpu)->nr_uninterruptible;
 *     avenrun[n] = avenrun[0] *
 *                  exp_n + nr_active *
 *                  (1 - exp_n)

Real life makes the code somewhat complex: per-CPU counters, tickless kernels, hotswap CPUs, lack of floaing point code requiring a fixed-point implementation of exp(n). But it’s easy to see that these are all working towards faithfully implementing the method described in the comment.

You’ll note that Linux counts threads, not just processes, which answers your question.

Method 4

The 3 load averages are something like a logarithmic function revolving around the number 1. Something similar to f(x) = eX (e to the X exponent). Technically a fixed-point representation of a exponential decay function simulating a average. They are additive, per CPU, so a full load might look like 4.00 on a quad-core system. The first number is the average over the last minute, the second is the average over the last five minutes, and the third is the average over the last 15 minutes. I thought an answer should be dropped here that mentions that.

Method 5

Instant load: number of tasks running or waiting to run, or in another way, the number of tasks willing to run

Load average: the measure above but exponentially averaged with previous samples of the same measure

Both of these numbers are unbounded, and often much larger than N.

To be clear: the load count on Linux includes threads, there is no doubt about it. You can produce an arbitrarily large load with a single process that creates many threads.