High Linux Load, Low CPU usage: When CPU Throttling Increased CPU Load

TL;DR

A highly concurrent workload can create a large population of runnable tasks, and when that workload is also constrained by a CPU limit, those tasks may take longer to accumulate the CPU time they need while competing with everything else on the host. On an already contentious node, that deeper runnable population can increase scheduling and context-switching overhead, push up system CPU time, and contribute to a high load average even when overall CPU utilisation still suggests that plenty of physical capacity is available.

The subtle part is that spare CPU on the host does not necessarily mean every workload is free to use it, because the Linux scheduler is sharing execution time across runnable work while cgroup CPU limits can independently restrict how much CPU a particular workload is allowed to consume.

This blog post discusses exactly one such instance in detail, so if you feel intrigued enough read on!

There are already plenty of blogs explaining Linux load average, and although I am not convinced there is necessarily a room for another one, especially in the AI era where you can ask a tool what a number means and get a decent explanation in seconds, I am writing this one anyways - partly because the incident underneath turned out to be an interesting one, and it might help a 'future-me".

Background

The load average on all 4 worker nodes in one of our dev k8s clusters had climbed a little above 90 and stayed there long enough to create some some dev. unrest. This was a relatively small cluster, with only 4 worker nodes in the data plane, but each node was a 112-core machine, and CPU utilisation was still sitting below 40%. So there was no obvious shortage of physical CPU capacity, and yet Linux was reporting a load number that made the nodes look far busier than the CPU graphs suggested.

That combination is what made the incident interesting, because my 1st instinct was still the familiar one, high load must mean CPU pressure, but if that were true, why were most of the processors idle?

The answer starts with a detail in how Linux calculates load that is easy to forget.

Load Average != CPU Utilisation

On Linux, load average is broader than a count of processes waiting for CPU. It includes tasks that are runnable, along with tasks in uninterruptible sleep, commonly shown as D state, so a high load average does not automatically mean the CPUs themselves are saturated.

So you may say that a node can have a high load because lots of work is ready to run, and/or because tasks are stuck in kernel waits - or both are happening at the same time, while the CPU utilisation graph can still look fairly ordinary.

First, check the boring explanation

The obvious place to start was uninterruptible sleep. Disk IO problems are a common reason tasks accumulate in D state, so I checked the wa figure in top, looked directly for processes in D, and then used iostat -x to see what the storage devices themselves were doing.

Nothing stood out.

iowait was close to 0, there were almost no processes sitting in D, and neither device latency nor utilisation suggested a storage queue building underneath us. None of those measurements individually proves that I/O is healthy, but taken together they made storage a much less convincing explanation.

That left the other half of the load calculation, runnable work.

The question was now why the machine appeared to have a substantial runnable population while using only a fraction of its physical CPU capacity.

System CPU provided a clue

Total CPU utilisation hides quite a bit, so the next useful split was user time versus system time. User time is primarily the application executing its own instructions, while system time represents work being performed in the kernel on behalf of those applications, including syscalls, scheduling activity, context switching and other kernel overhead.

User CPU was relatively quiet, but system CPU was disproportionately visible.

That did not diagnose the issue by itself, but it gave me somewhere useful to look next, and vmstat added another clue. Context switching was running very high, and it also pointed towards more runnable work than I expected from a machine that otherwise looked mostly idle.

This is where the scheduler behaviour starts to matter.

A runnable process is not necessarily a process that is getting useful CPU time right now, it only means the process is ready to run. As the number of runnable processes increases, all of them have to share the same finite amount of CPU execution time, and the scheduler can still be doing exactly what it is supposed to do while individual processes make slower progress.

If 1 task needs 100 ms of actual CPU execution to complete a piece of work, it still needs roughly those same 100 ms, but if many other runnable tasks are competing for the same CPUs, it may take much longer in wall-clock time to accumulate them because it repeatedly waits while other tasks get their turn.

fair scheduling only means runnable work gets a proportional share of execution time - Fair does not mean fast.

At that point I wanted to know what kind of switching was happening.

pidstat -w is useful here because it separates voluntary context switches, where a task gives up the processor because it cannot continue, from involuntary switches, where the scheduler takes the processor away from a task that can still run. A high involuntary rate can support a scheduling-pressure or CPU-contention hypothesis, although it is not enough by itself to prove one.

$ pidstat -w 1

19:42:11      UID       PID   cswch/s   nvcswch/s  Command
19:42:12        0      18241      8.00       312.00  nginx
19:42:12        0      18267     11.00       287.00  nginx
19:42:12     1000      24109    426.00         3.00  worker

In this case, the pattern was dominated enough by involuntary switching that the scheduler became much more interesting than the disks.

A signal was hiding nearby

Upon further investigation a security related workload, Akto, was showing severe CPU throttling in Grafana.

K8s CPU limits are enforced through cgroup CPU bandwidth control, where a workload gets a quota of CPU time for a scheduling period, and once it consumes that quota, the kernel prevents it from executing again until the quota cycle refreshes - which means a highly concurrent workload may still have many tasks ready to execute, but once its cgroup exhausts its CPU quota, that runnable work cannot actually receive CPU time until the quota is replenished.

So you can end up with those runnable tasks sitting idly in the runnable queue piling up while the host itself still has physical CPU sitting idle, inflating the load numbers.

I would not use the throttling observation alone to explain the entire load average of 90, but it mattered because it demonstrated something that is easy to miss when looking at node-level dashboards, physical CPU availability and workload CPU availability are not the same thing.

That turned out to be the more useful lesson from the incident.

Then k8s made the problem stranger

Once I started tracing the runnable work back to the workloads on the node, the ingress layer immediately looked suspicious.

We were running 20 ingress-nginx replicas across only 4 workers, which meant roughly 5 controller pods could land on the same node, and each controller was configured with 48 nginx worker processes. That gave us around 240 nginx workers associated with the ingress layer on a single machine.

There is an important qualification here, 240 nginx workers does not mean 240 workers are constantly runnable and competing for CPU. NGINX is event-driven, and many of those workers may be blocked waiting for network events rather than consuming execution time.

The configuration was still worth questioning though, because ingress-nginx uses worker-processes: auto by default, which sizes the worker population according to the CPUs it considers available. In a k8s environment, that can become interesting when several ingress replicas share the same large host, because you can end up replicating a worker model that was sized with the host's CPU topology in mind several times over on the same machine.

The point was not simply that there were too many nginx processes. The point was that the scheduler was already telling us there was a large runnable population and heavy preemption activity, and the ingress deployment gave us an obvious place to investigate where some of that runnable work might be coming from.

The troubleshooting model I kept

What came out of this was less a magic command than an order in which to ask questions.

The important part is that load average becomes the beginning of the investigation rather than its conclusion.

A large number tells you Linux sees work it considers active, but it does not tell you whether the bottleneck is CPU execution, an uninterruptible kernel wait, scheduler pressure, a cgroup policy, or some combination of them.

In our case, the practical changes were fairly mundane once the shape of the problem became clearer. We revisited the nginx worker count instead of blindly allowing each ingress instance to size itself from the CPUs it could see, questioned why 20 ingress replicas were concentrated across only 4 workers, and revisited CPU limits on the heavily throttled workload.

So in a nutshell the key takeaway is that the scheduler can be perfectly fair and the application can still get slower, because fairness means sharing execution time, and the more runnable work there is to share it with, the longer each task can take to accumulate enough CPU time to do something useful. Add k8s CPU limits on top of that and the picture gets even stranger, because a host can have physical CPU sitting idle while a particular workload is still not allowed to use it.

A load average around 90 beside 40% CPU initially looked like 2 monitoring signals disagreeing with each other, but they were really exposing different layers of the same system.

The CPU could be idle, but that does not mean the workload could have it.