# When Load Shedding Looks Like Failure

![](https://cdn.hashnode.com/uploads/covers/6a25ff63980616b1c020fc7a/16d68f6e-9b2a-4520-820b-6658bfccfaf7.png align="center")

## Issue?

Last week I read through a post-mortem for an incident where a microservice took itself out by doing exactly what it was designed to do.

The setup was ordinary. Service A made synchronous HTTP calls to Service B, and during a traffic surge Service B hit its concurrency ceiling of 200 active requests. Rather than let its threads lock up it shed the excess, rejecting new requests with a 503. From Service B's side that worked fine: CPU stayed flat, and it kept serving the requests it had already accepted without degrading.

The break was upstream. Service A's circuit breaker judged health by the share of 5xx responses over a rolling window, and to a client-side health checker a 503 is a 503, with nothing in it that separates a service shedding load because it's busy from one whose dependencies have collapsed underneath it. As the surge pushed the shed responses past the error-rate threshold the circuit tripped, and Service A ejected Service B from rotation, cutting off all traffic including the 200 concurrent requests it was still comfortably handling.

That pushed the remaining traffic onto the surviving pods, which hit their own concurrency limits almost immediately, shed load, returned more 503s, and got ejected in turn. The load shedder doing its job correctly is what pulled the healthy fleet out of rotation, one pod at a time.

## Fix

The fix was a status code. We configured the shedder to return 429 instead of 503, since client-side circuit breakers generally ignore 4xx when counting server health failures. A 429 says you're sending too much where a 503 says I'm broken, and that difference is what decouples being busy from being ejected.

The client side had to change with it. On a 429 Service A backs off exponentially with jitter rather than counting the response against the breaker, so the retries spread out and Service B gets a window to drain its queue instead of taking the whole surge again the moment the timer expires.