# Cache Stampede - Outage from Concurrency

[**Caching**](https://en.wikipedia.org/wiki/Caching) usually enters a system for a simple reason: the backend is doing work that does not need to be repeated for every request. A value that is expensive to retrieve or calculate can be stored in [**Redis**](https://redis.io/docs/latest/), local memory, or an HTTP proxy cache. The next caller gets the saved result instead of repeating the database query or upstream request.

That arrangement works well while the cache is warm. The more interesting behaviour begins when a popular value disappears.

Several requests can reach the cache within the same few milliseconds, find the same missing key, and follow the normal fallback path. Each request is behaving correctly on its own. Together, they can send a burst of duplicate work to the database or another service. That burst is commonly called a **cache stampede**.

Before getting into the stampede itself, it helps to place it among the common caching patterns.

## Common caching patterns

| Pattern | Read behaviour | Write behaviour | Where the logic usually lives |
| --- | --- | --- | --- |
| Cache-aside | The application checks the cache, then loads from the database after a miss | The application updates the database and usually invalidates the cache | Application |
| Read-through | The application asks the cache layer for the value; the cache loads it after a miss | Depends on the write pattern | Cache library or data layer |
| Write-through | Reads usually come from the cache | The cache writes to the database before returning success | Cache or data layer |
| Write-behind | Reads come from the cache | The cache accepts the write and persists it later | Cache layer and worker |
| Write-around | Reads use cache-aside or read-through | Writes go directly to the database and usually invalidate the cache | Application |
| Refresh-ahead | Reads use the cached value | A worker refreshes entries before expiry | Cache layer or background process |

## Cache Aside

![](https://cdn.hashnode.com/uploads/covers/6a25ff63980616b1c020fc7a/a653e7f7-1b68-4b4d-a7ca-aef4e6637d90.png align="center")

Cache-aside is common with Redis. The application owns the read path, handles misses, loads from the database, and stores the value back in the cache. That direct control is useful, though it also means the application must decide what happens when many callers miss the same key at once.

## Read Through

![](https://cdn.hashnode.com/uploads/covers/6a25ff63980616b1c020fc7a/3d1e2509-1a2b-423d-9de6-ba3086e61365.png align="center")

## Write Through

![](https://cdn.hashnode.com/uploads/covers/6a25ff63980616b1c020fc7a/6bfabdef-d50a-47f1-8125-a293d999f062.png align="center")

## Write Behind

![](https://cdn.hashnode.com/uploads/covers/6a25ff63980616b1c020fc7a/63eca22c-0ffc-4bf2-b66c-1a2cdb375d10.png align="center")

## How a cache stampede starts

Consider a product endpoint receiving several thousand requests per second. Product details stay in Redis for five minutes. During that period, nearly every request is cheap because the database is rarely touched.

When the [**TTL**](https://redis.io/docs/latest/commands/ttl/) ends, the entry disappears. Several requests may observe the miss before any one of them has rebuilt the value.

![](https://cdn.hashnode.com/uploads/covers/6a25ff63980616b1c020fc7a/3a9d5200-dfea-4f7a-87d7-ef2fddbe05b0.png align="center")

The problem is visible in the duplicate backend work. One expired key can turn into hundreds of identical queries. If the value comes from an internal API, the burst moves there instead. As the backend slows down, requests stay active for longer, connection pools begin to fill, and timeouts appear. Retries can add another wave of traffic before the first one has cleared.

A cache outage creates a larger version of the same failure. Instead of one popular key expiring, many keys miss at once and most application traffic falls through to the database. A fallback path that works for occasional misses may collapse when it becomes the main path.

## Request coalescing

Request coalescing groups concurrent requests for the same key around one backend operation. One caller performs the load while the others wait for the same result.

![](https://cdn.hashnode.com/uploads/covers/6a25ff63980616b1c020fc7a/300d3449-f9a7-44e4-8e10-d8c92a46e435.png align="center")

The key is what defines the group. Requests for `product:123` can share one load, while requests for `product:456` continue independently. This removes duplicate work without serializing unrelated requests.

The coalescing logic can live inside the application or at an HTTP caching proxy. Both approaches reduce repeated work, though they operate at different layers and have different context available to them.

## Coalescing inside a [**Go**](https://go.dev/) service

Go applications often use [`golang.org/x/sync/singleflight`](https://pkg.go.dev/golang.org/x/sync/singleflight). A `singleflight.Group` tracks active function calls by key. When another caller requests the same key, it waits for the existing call and receives the same value or error.

A simplified cache-aside implementation might looks like this

```go
type ProductService struct {
	cache Cache
	db    ProductRepository
	group singleflight.Group
}

func (s *ProductService) GetProduct(
	ctx context.Context,
	id string,
) (*Product, error) {
	key := "product:" + id
	product, err := s.cache.Get(ctx, key)
	if err == nil {
		return product, nil
	}
	value, err, _ := s.group.Do(key, func() (any, error) {
		product, err := s.cache.Get(ctx, key)
		if err == nil {
			return product, nil
		}

		product, err = s.db.GetProduct(ctx, id)
		if err != nil {
			return nil, err
		}

		if err := s.cache.Set(
			ctx,
			key,
			product,
			5*time.Minute,
		); err != nil {
			// Record the cache error while returning
		}

		return product, nil
	})
	if err != nil {
		return nil, err
	}

	return value.(*Product), nil
}
```

The first cache lookup stays outside `singleflight`, keeping the common hit path cheap. After a miss, callers join the group using the cache key. The cache is checked again inside the grouped function because another process or request may have populated Redis during the gap.

![](https://cdn.hashnode.com/uploads/covers/6a25ff63980616b1c020fc7a/a8c1dfe2-f38c-4e1e-a4e4-2ed5cb4b8842.png align="center")

The diagram already shows the process boundary and the one-in-flight-per-key behaviour, so the surrounding prose should focus on the limits.

A `singleflight.Group` lives in one process. In a [**Kubernetes**](https://kubernetes.io/docs/concepts/workloads/pods/) deployment with twenty application pods, each pod has its own group and may start its own load for the same key. That can still reduce thousands of database calls to a few dozen, which is a meaningful improvement. Reducing the operation to one cluster-wide load requires another coordination mechanism, such as a [**distributed lock**](https://redis.io/docs/latest/develop/clients/patterns/distributed-locks/) or a shared cache-filling service.

The application also needs a policy for waiting callers. They may block, receive stale data, or fail after a timeout. A lock or coalescing primitive only decides who performs the load. It does not decide what everyone else should experience.

Errors need similar care. `singleflight` shares the leader’s error with the waiting callers. If several hundred callers receive the same timeout and retry immediately, the next burst can recreate the stampede. Backoff, retry jitter, and stale fallback help stop those retries from lining up again.

Per-key coalescing also offers little protection when every request uses a different key. A cold cache can still create thousands of unrelated backend loads. A separate concurrency limit around the fallback path is useful in that case.

## [**Python** `asyncio`](https://docs.python.org/3/library/asyncio-task.html) equivalent

```python
class SingleFlight:
    def __init__(self) -> None:
        self._lock = asyncio.Lock()
        self._inflight = {}

    async def do(self, key, loader):
        async with self._lock:
            task = self._inflight.get(key)

            if task is None:
                task = asyncio.create_task(loader())
                self._inflight[key] = task

        try:
            return await asyncio.shield(task)
        finally:
            if task.done():
                async with self._lock:
                    if self._inflight.get(key) is task:
                        self._inflight.pop(key, None)
```

The lock protects the dictionary of active tasks. The loader runs outside it, so unrelated keys can continue independently. A production implementation would need careful handling for failed tasks, cancellation, cleanup, and memory limits.

Application-level coalescing fits well when the expensive work includes database queries, internal service calls, permission checks, or business logic that an HTTP proxy cannot safely understand.

## Coalescing at [**NGINX**](https://nginx.org/en/docs/http/ngx_http_proxy_module.html)

NGINX can coalesce requests at the HTTP cache layer with [`proxy_cache_lock`](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_cache_lock).

```nginx
proxy_cache_path /var/cache/nginx
    keys_zone=api_cache:100m
    max_size=10g
    inactive=30m;

server {
    location /public/catalog/ {
        proxy_pass http://catalog_service;

        proxy_cache api_cache;
        proxy_cache_valid 200 5m;

        proxy_cache_lock on;
        proxy_cache_lock_timeout 3s;
    }
}
```

![](https://cdn.hashnode.com/uploads/covers/6a25ff63980616b1c020fc7a/d8f3ca5d-8946-46b0-9d6d-fadcd33b6c49.png align="center")

The cache key is central to this design. If the response changes according to the tenant, authorization header, language, cookies, or query parameters, the cache rules must account for those differences. Some authenticated responses should not be stored in a shared proxy cache at all.

`proxy_cache_lock_timeout` controls how long a request waits behind the active cache fill. A longer timeout protects the origin more aggressively, though callers may wait behind a slow request. A shorter timeout reduces the waiting period while allowing more requests to reach the application.

For expired entries, NGINX can serve stale content while a background request refreshes the cache:

```nginx
proxy_cache_use_stale updating;
proxy_cache_background_update on;
```

This works well for content that can safely be a little old, such as public documentation or catalogue data. Permissions, balances, and fast-changing inventory often need stricter freshness.

## Application or proxy?

| Question | Application coalescing | NGINX cache locking |
| --- | --- | --- |
| What is grouped? | Any operation identified by an application key | HTTP requests sharing a cache key |
| Can it directly protect database or API work? | Yes | Indirectly |
| Does it understand tenant and user context? | Yes | Only through cache configuration |
| Is the result cached automatically? | No | Yes, when cacheable |
| Typical scope | One application process | One NGINX cache instance |
| Best fit | Internal and business-aware work | Safely cacheable HTTP responses |

Both approaches can exist in the same request path.

![](https://cdn.hashnode.com/uploads/covers/6a25ff63980616b1c020fc7a/366c45f7-6f68-4eee-8ea9-8e60017d802c.png align="center")

The proxy reduces repeated HTTP requests before they reach the service. The application then coalesces duplicate backend work after a Redis miss. This is useful when each layer has a clear purpose and its own freshness rules are visible through metrics and logs.

Layering several caches becomes harder to operate when TTLs and invalidation rules drift apart. During an incident, engineers need to know which layer returned the value, how old it was, and why it was kept or refreshed.

## Coalescing is only part of the fix

Request coalescing addresses simultaneous work for the same key. A production cache design usually needs a few other controls.

TTL jitter spreads expiry over time so large groups of entries do not disappear together. A concurrency limit around the fallback path protects the backend during a cold-cache event with many different keys. [**Stale-while-revalidate**](https://www.rfc-editor.org/rfc/rfc5861.html) keeps routine refreshes away from the critical request path when slightly old data is acceptable.

A full cache outage also needs a deliberate fallback policy. Sending every request directly to the database can turn a Redis incident into a database incident. Local stale data, strict fallback limits, or temporary load shedding may be safer than an unrestricted bypass.

## Closing thought

A cache stampede begins with normal request behaviour: a value is missing, so callers try to rebuild it. The failure comes from concurrency.

The right layer depends on where the necessary context exists. Public HTTP responses often fit a proxy cache. Database queries and business-aware operations usually belong closer to the application. Some systems need both, along with limits on total fallback work and a clear decision about when stale data may be served.