I was cleaning up some old notes recently and came across one I'd written a few years ago while trying to understand Linux process accounting.
Back then I was doing a lot of perf-related work and benchmarking, and I remember being confused by something I'd seen countless times:
$ time ./my_program
real 2.1s
user 6.4s
sys 0.3s
How could a program that finished in just over 2 seconds spend 6 seconds on the CPU?
I dug into it at the time and made a few notes. Reading them again was a nice refresher, so I cleaned them up a bit and thought I'd share them here. Maybe they'll save someone else a trip down the same rabbit hole, or at the very least serve as a refresher to aa future me :-)
What does time actually measure?
The linux time command prints three values:
real
user
sys
I used to look at them without thinking much about what each one actually meant.
real
This one's straightforward as It's just the wall-clock time.
start the program -> RUN -> stop the program.
And whatever a stopwatch would show is real.
It includes everything:
CPU execution
waiting on disk
waiting on the network
sleeping
scheduler delays
lock contention
Basically, the total time the process existed.
user
This is where things started making more sense.
user is the amount of CPU time spent executing your application's code.
Think any runtrime like Go, Rust, Python, Java, C++, etc. But do notice I said CPU time, not elapsed time.
sys
sys is also CPU time, but it's time spent inside the Linux kernel on behalf of your process.
Typical examples are:
read()
write()
send()
recv()
page faults
filesystem work
So why can user be larger than real?
This was the part that finally made it click for me.
Imagine your machine has four CPU cores.
Your program spins up two worker threads and both stay busy for two seconds.
Core 0 ████████ 2 sec
Core 1 ████████ 2 sec
Core 2 idle
Core 3 idle
The program still finishes after two seconds.
So:
real = 2 sec
But Linux accounts CPU usage separately for each core.
Core 0 = 2 sec
Core 1 = 2 sec
Total CPU time = 4 sec
Which means this is perfectly valid:
real 2 sec
user 4 sec
CPU time is accumulated across every CPU executing your process.
What about everything else running on the machine?
Another question I had was whether time includes CPU used by other applications.
Suppose your workstation has 32 cores.
Chrome is open.
Docker is busy.
Maybe a few k8s clusters are running locally.
None of that matters.
time only reports CPU consumed by your process (and usually any child processes that it creates and waits for).
Everything else is ignored.
Python's process_time()
Python has something very similar.
import time
start = time.process_time()
do_work()
end = time.process_time()
This measures CPU time for the current process.
For example:
time.sleep(5)
will roughly look like this:
Wall time ≈ 5 sec
CPU time ≈ 0 sec
because the process is sleeping, not executing instructions.The same idea applies to network I/O. f an HTTP request takes 500 ms but your process only spends 4 ms actually executing code, then:
Wall time = 500 ms
CPU time = 4 ms
Which timer should I use?
Use the timer that matches the question you're asking. For example, if you're trying to benchmark how much CPU your code consumes:
time.process_time()
makes sense.
If you're measuring request latency, API response times, or anything the user experiences:
time.perf_counter()
is usually the better choice as they're measuring two different things.
Final thoughts
Looking back, this wasn't a particularly difficult concept. I just never had a reason to stop and think about what Linux was actually reporting.
It's funny how some tools become muscle memory. we use them almost every day, but don't always understand what they're doing underneath.
Anyway, I figured these old notes were worth keeping around. If you've ever looked at output like this:
real 2.0
user 7.3
sys 0.2
and wondered if something was broken, it probably isn't.
It usually just means your program managed to keep multiple CPU cores busy at the same time.
