Reading Latency Effectively
The Median Vs. the Mean, and a breakdown of what each percentile Is attempting to tell you
TL;DR
Mean latency describes total latency averaged across requests.
p50 describes the middle of the distribution.
p90 and p99 describe progressively higher boundaries in the tail.
The latency SLO measures how many requests crossed the threshold that matters to the service objective.
Taken together, they provide considerably more information than a single average latency panel, while still leaving each metric to describe the property it was actually designed to measure.
From what I have experienced most app. observability dashboards start with an average because it is familiar and easy to calculate. If an exporter gives you the usual histogram _sum and _count metrics, rate(sum) / rate(count) gives you mean latency, but the number it produces is not necessarily representative of what a typical request experienced.
Latency distributions are usually right-skewed. Most requests might complete within a relatively narrow range while a small number take significantly longer, and those slower requests can pull the mean away from the centre of the distribution.
Percentiles describe that distribution differently, and looking at several of them together gives more information than either the mean or a single percentile on its own.
p50 and the mean
Take a simple example where 99 requests complete in 10 ms and 1 request takes 5 seconds.
The median, or p50, is 10 ms. The mean is roughly 60 ms, even though no request in the sample actually took 60 ms, because the single 5-second request contributes enough latency to pull the average upward.
If the 5-second request becomes a 30-second request, the p50 remains 10 ms because the middle of the distribution has not changed, while the mean rises significantly.
This makes p50 useful for describing the latency around the middle of the request population.
During an incident mean latency can increase because a large proportion of requests became moderately slower, or because a much smaller number became extremely slow, and the mean alone cannot distinguish between those 2 cases. Thus a simultaneous view of the mean and percentiles gives more context about where the change occurred.
What each percentile is actually expressing
The simplest way to read percentiles is as boundaries within the latency distribution.
p50 represents the middle of the distribution. If p50 moves materially, which is consistent with causes that affect common request paths or shared resources, such as CPU pressure across multiple replicas, a database dependency becoming slower, connection pool pressure, a cache losing effectiveness, an additional operation in the common request path, or an increase in workload.
A p50 increase does not mean that every request became slower, but it does mean that the midpoint of the distribution has moved rather than the change being confined entirely to a small tail.
p90 describes a boundary further into the distribution. If p90 increases while p50 remains relatively stable, the slower part of the workload has changed while the middle has not moved to the same degree.
That pattern can appear when different categories of requests behave differently. Cache hits may remain fast while cache misses become slower, 1 route may perform more work than others, a tenant may generate heavier requests, or a subset of operations may depend on storage or another slower downstream path.
p99 sits much further into the tail. Changes here can expose behaviour that occurs relatively infrequently, including an unhealthy replica that is still receiving traffic, occasional garbage collection pauses, lock contention, retries, slow shards, hot keys, or intermittent downstream latency.
The p99 value itself is not "the slowest 1%." It is the latency boundary at or below which approximately 99% of the observations fall, leaving roughly 1% beyond it.
Read the gap, not just the numbers
A p99 of 500 ms means something different when p50 is 450 ms than when p50 is 20 ms.
In the first case, latency is relatively high across much of the distribution. In the second, the centre remains fast while the tail extends much further to the right. The p99 number is identical, but the underlying distributions are different.
When p50, p90, and p99 all rise together, the distribution has broadly shifted to the right. Changes in shared dependencies, CPU or memory pressure, database latency, common request paths, deployments, network behaviour, or workload levels can produce this shape.
When p50 remains stable while p90 and p99 rise, the change is concentrated further into the distribution. Breaking the metric down by route, tenant, operation, shard, availability zone, replica, or another relevant dimension can show whether a particular class of traffic has become slower.
When p50 and p90 remain relatively stable while p99 rises, the change is concentrated in the far tail. Replica-specific problems, GC pauses, retries, lock contention, hot keys, overloaded shards, intermittent dependencies, and unusually expensive request paths are examples of conditions that can produce that pattern.
A related pattern appears as systems approach saturation. Queueing delay tends to rise non-linearly as utilisation increases, which means the slower requests can begin accumulating additional waiting time before the median changes significantly. A thread pool approaching saturation, a connection pool running short, a busy disk, or a downstream service moving closer to its capacity limit can therefore show movement in p99, followed by p90 and eventually p50 if the pressure continues.
That tail-first behaviour is not universal though as a change applied to the normal request path can move the whole distribution immediately. An additional synchronous dependency call, a slower query used by every request, a disabled cache, or a configuration change affecting all replicas can shift p50, p90, and p99 at roughly the same time.
Prometheus histograms
When these percentiles come from Prometheus histograms, histogram_quantile() is estimating a percentile from bucket counts rather than calculating it from the original request latency values.
A classic Prometheus histogram records how many observations fall below each configured bucket boundary. Once an observation has been recorded, Prometheus knows which bucket contains it, but it does not retain the exact latency value as part of the histogram.
Consider a histogram with finite bucket boundaries at:
100 ms, 200 ms, 500 ms, 1 s
followed by the required +Inf bucket.
If the requested percentile falls somewhere between 500 ms and 1 second, Prometheus knows the number of observations that landed within that range but not their exact positions. histogram_quantile() therefore apparently estimates the quantile within the bucket.
Low traffic requires recalibration
Percentiles also depend on the number of observations in the calculation window.
For an internal API receiving only a few calls each minute, the same 5-minute p99 is based on a much smaller sample. A single slow request entering or leaving the query window can cause a large change in the reported percentile.
For very low-volume workloads, traces, or structured request logs can provide more direct information about individual slow requests than a high percentile calculated from a small sample.
Latency SLOs
A latency SLO normally measures the proportion of valid requests that completed within a defined threshold.
For example, an SLO might require:
99% of valid requests to complete within 500 ms.
Requests above 500 ms then count as bad events for that latency objective, regardless of whether the current p99 happens to be 510 ms or 2 seconds.
Error-budget burn rate shows how quickly requests exceeding the latency objective are consuming the available budget. Percentiles show how the shape of the latency distribution has changed. Additional dimensions such as route, tenant, replica, zone, shard, or dependency can then identify where the slower population is concentrated, while traces, logs, and resource metrics provide more detail about the individual requests and components involved.
