# Stop Pinging Kafka - A Zero-Cost Health Check Pattern (and What Can Go Wrong)

Some teams may assess Kafka's health incorrectly.

They publish test messages to a topic. Or they spin up an AdminClient just to call `DescribeClusterAsync()`. Or they assume "no errors = healthy" and hope for the best. All of these either waste resources, pollute your topics, or tell you nothing about the connections your producers and consumers actually use.

There's a better pattern. But it has sharp edges, and if you're not careful, it can make things worse than having no health check at all.

## **The Problem**

Kafka isn't a request-response system. You don't ping it. Your producers and consumers maintain long-lived connections that multiplex heartbeats, fetches, and metadata requests over the same socket. So the question "is Kafka healthy?" doesn't have a clean answer — what you're really asking is "can my client reach the broker right now?"

The common approaches miss this:

**Publishing a test message** pollutes production topics, consumes broker resources, and raises security concerns (does your producer even have permission to write to a health-check topic?).

**AdminClient metadata requests** spin up a separate connection with its own lifecycle. You're now managing an extra client just to ask "are you alive?" — and the answer only proves that *that* connection works, not the ones your actual producers and consumers use.

**Implicit "no error = healthy"** is the worst. A producer that hasn't sent anything recently won't surface a broken connection until the next produce call fails. By then, you're already in trouble.

## **The Pattern: Piggyback on Statistics**

If you're using librdkafka (and if you're on .NET with confluent-kafka-dotnet, you are), your client already has a built-in health signal. librdkafka emits a statistics JSON blob at a configurable interval — [`statistics.interval.ms`](http://statistics.interval.ms). In .NET, you hook into it with `SetStatisticsHandler`.

The idea is simple: if the callback fires, your client's I/O loop is alive. If it stops firing, something's wrong. Zero extra connections, zero extra requests, and it reflects the actual connections your producers and consumers use.

**csharp**

```csharp
var config = new ConsumerConfig
{
    BootstrapServers = "broker:9092",
    StatisticsIntervalMs = 5000,
};

private long _lastStatsTick = Environment.TickCount64;

using var consumer = new ConsumerBuilder<string, string>(config)
    .SetStatisticsHandler((_, _) =>
    {
        Interlocked.Exchange(ref _lastStatsTick, Environment.TickCount64);
    })
    .Build();

app.MapGet("/health/kafka", () =>
{
    var elapsed = Environment.TickCount64 - Interlocked.Read(ref _lastStatsTick);
    return elapsed > 10_000
        ? Results.Problem("Kafka stats stale", statusCode: 503)
        : Results.Ok("Kafka healthy");
});
```

The callback fires on its own schedule. The kubelet hits your endpoint on its own schedule. The two are completely decoupled.

## **Where the Callback Lands**

This is the part most people skip, and it matters.

The statistics callback fires on a **librdkafka internal thread** — a native C thread that manages all of librdkafka's I/O work. Not a .NET ThreadPool thread. Not the thread calling `Consume()`. A native thread that temporarily enters the CLR to execute your delegate.

This means three things:

**Don't block in the callback.** The thread calling your stats handler is the same thread doing broker heartbeats, fetch loops, and produce delivery. Block it for two seconds parsing JSON or writing to a database, and you've stalled librdkafka's entire event loop. Brokers may think the connection is dead. Consumers may get kicked from the group.

**Don't call back into the same client.** Calling `producer.ProduceAsync()` from inside the stats callback can deadlock — you're asking librdkafka to do work while it's already inside a callback on its own thread.

**Be thread-safe.** Any shared state you touch from the callback needs synchronization, because your application code runs on different threads. Use lock-free primitives where possible. [`Interlocked.Exchange`](http://Interlocked.Exchange) on a `long` timestamp is cheap and sufficient for a basic health check.

If you want to parse the stats JSON (and you should — more on that later), enqueue the raw string and process it on a background task:

**csharp**

```csharp
private readonly Channel<string> _statsChannel =
    Channel.CreateUnbounded<string>();

.SetStatisticsHandler((_, json) =>
{
    _statsChannel.Writer.TryWrite(json);
})

async Task ProcessStatsAsync(CancellationToken ct)
{
    await foreach (var json in _statsChannel.Reader.ReadAllAsync(ct))
    {
        var stats = JsonSerializer.Deserialize<StatsPayload>(json);
        _metricsService.Record(stats);
    }
}
```

## **You Can't Trigger It On Demand**

There's no API to force the stats callback to fire. No `GetStatsNow()`, no `TriggerStatistics()`. It fires when librdkafka's I/O loop decides the interval has elapsed.

But you don't need to. The pattern isn't "kubelet triggers callback → callback returns health." It's "callback fires on its own cadence and writes to shared state → kubelet hits your endpoint → endpoint reads that shared state."

The tradeoff: your health answer is always slightly stale — at most one [`statistics.interval.ms`](http://statistics.interval.ms) old. If you set the interval to 5 seconds, your endpoint is answering "was Kafka healthy as of up to 5 seconds ago?" not "is Kafka healthy right now this instant?"

For Kubernetes liveness and readiness probes, that's almost always fine. The kubelet doesn't expect a real-time answer — it expects a recent-enough one. Its failure thresholds and retry logic already absorb that gap.

If you truly need on-demand connectivity verification at the exact moment someone asks, this passive pattern won't do it. You'd need an active probe (AdminClient metadata request). But in practice, most probe scenarios don't need that level of immediacy.

## **How This Can Make Things Worse**

Here's where I'd push back on the "just use the stats callback" advice. This pattern has real failure modes, and they're not obvious.

### **The callback fires even when the broker is unreachable**

This is the most dangerous one. librdkafka's I/O loop keeps running even when it can't connect to any broker — it's busy retrying. The stats timer is tied to that I/O loop, not to connection state. So the callback **keeps firing** while disconnected.

If your health check only asks "did the callback fire recently?", it reports healthy while Kafka is actually unreachable. The stats JSON itself would show brokers as `DOWN` or `CONNECTING`, but you'd never know unless you parse it — which you were told to keep out of the callback.

This is the gap between "librdkafka's I/O thread is alive" and "I can actually reach Kafka." They're not the same thing.

### **GC pauses cause false unhealthy → restart storms**

A long Gen 2 GC pause in .NET freezes the managed side of the callback. The native librdkafka thread can't transition into the CLR, so the timestamp goes stale. Your health endpoint reports unhealthy. The kubelet restarts the pod.

But Kafka was fine — the process was just collecting garbage. Now you've restarted a pod that didn't need restarting, triggered a consumer group rebalance (lag spike for all consumers in the group), potentially caused duplicate message processing (offsets not committed before kill), and not fixed the root cause. If GC pressure was workload-driven, the new pod hits the same pattern and fails again.

This is a restart loop driven by your own health check.

### **Blocking the callback stalls the very thing it monitors**

If someone puts expensive work in the callback — JSON parsing, logging to a database, calling a metrics API — they block librdkafka's I/O thread. That thread is also doing heartbeats, fetches, and produce delivery. Blocking it causes brokers to kick the consumer out of the group (missed heartbeats), produce timeouts, and the stats callback itself going silent. The health check creates the problem it's supposed to detect.

### **Shutdown races cause message loss**

During graceful shutdown, librdkafka drains and closes connections. The stats callback stops firing or reports disconnecting states. If your health check doesn't gate on a shutdown flag, it reports unhealthy mid-shutdown. The kubelet sees unhealthy and forcefully kills the pod — bypassing graceful shutdown. In-flight messages aren't acknowledged. Consumer offsets aren't committed. You lose data because of a race between your health check and your shutdown sequence.

### **Flapping and thundering herds**

If the grace period is too tight, the probe flaps between healthy and unhealthy. The load balancer keeps adding and removing the pod. Clients get connection resets, retries pile up, and the remaining healthy pods absorb the extra load — potentially causing them to fail too.

And if multiple pods share a trigger — a brief network blip, a shared workload spike causing GC pressure, a broker rolling restart — they can all fail health checks simultaneously. They all restart at once. Mass consumer group rebalances, broker connection storms, and lag spikes cascade across the system. The health check amplified a transient issue into a sustained outage.

## **Why Restarts Hurt: The Rebalance Problem**

Every one of those failure modes eventually leads to a pod restart. And when a Kafka consumer restarts, the consumer group rebalances.

A rebalance is Kafka's mechanism for reassigning partition ownership. It exists because Kafka's core guarantee is that every partition is consumed by exactly one consumer in the group at any time. When a consumer joins, leaves, or crashes, partitions that belonged to the departed consumer need a new owner.

The problem is that the classic rebalance protocol is stop-the-world. Every consumer in the group revokes all its partitions, the group leader reassigns everything, and then everyone resumes. Even consumers that were perfectly healthy and whose partitions didn't need to move.

So one pod restarts because of a GC pause, and every consumer in the group stops processing for the duration of the rebalance. Lag spikes across all partitions, not just the ones that needed reassignment.

## **How to Reduce the Blast Radius**

### **Cooperative rebalance protocol (KIP-429)**

Instead of revoking everything and reassigning, the cooperative protocol only revokes partitions that actually need to move. Consumers keep their current partitions and keep consuming while the coordinator figures out what needs to transfer.

In .NET:

**csharp**

```csharp
var config = new ConsumerConfig
{
    PartitionAssignmentStrategy = PartitionAssignmentStrategy.CooperativeSticky,
};
```

If consumer C dies, only its partitions get reassigned. Consumers A and B keep consuming without interruption. The blast radius shrinks from "everyone stops" to "two partitions briefly pause."

This has been available since Kafka 2.4. It's one config line. No broker upgrade needed. And yet most teams are still on the default eager protocol.

### **Static group membership (KIP-345)**

When a static member leaves (e.g., pod restart during deployment), the coordinator doesn't immediately rebalance. It waits up to [`session.timeout.ms`](http://session.timeout.ms) for that member to come back. If the consumer restarts within that window, it reclaims its original partitions without a full rebalance.

**csharp**

```csharp
var config = new ConsumerConfig
{
    GroupInstanceId = "consumer-A-unique",
    SessionTimeoutMs = 30000,
};
```

This is especially powerful for Kubernetes rolling deployments. If pods restart quickly, static membership can make deployments nearly invisible to the consumer group.

### **The new consumer group protocol (KIP-848)**

Available in Kafka 4.0+, this is a ground-up redesign. Broker-side coordination, fully incremental rebalancing natively. It's the long-term answer, but requires a newer Kafka version and a migration.

## **The Real Lesson**

Avoiding rebalances isn't the goal. Avoiding *unnecessary* ones is.

A necessary rebalance: a consumer genuinely died — OOM kill, segfault, network partition that won't recover. Its partitions need a new owner. You want that rebalance to happen. Suppress it, and those partitions go unconsumed forever.

An unnecessary rebalance: a consumer is perfectly healthy, but something made it look dead — a GC pause, a health check false positive, a brief network blip. The rebalance is a side effect of a false signal.

The strategies above reduce the *cost* of rebalances. But they don't eliminate the need for the signal itself. If a consumer is truly gone, you still need a rebalance.

The real fix for health-check-driven restart loops isn't "avoid rebalances." It's **don't restart healthy pods in the first place.** The rebalance is a symptom. The disease is the health check producing false positives.

## **Practical Takeaways**

*   Use the stats callback pattern. It's cheap, it's real, and it catches real problems.
    
*   Set a generous grace period — 2–3x the stats interval. If stats fire every 5 seconds, use 15 seconds of grace. This absorbs most GC pauses and jitter.
    
*   Don't use it as a liveness probe without understanding the GC interaction. Liveness failures cause restarts. A false positive here is destructive. Consider it for readiness only, or pair it with a high `failureThreshold`.
    
*   Parse the stats JSON off-thread to check actual broker connection state, not just "did the callback fire." If the callback fires but brokers show `DOWN`, that's a real Kafka problem. If the callback doesn't fire but the process is otherwise responsive, that's a runtime problem.
    
*   Gate the health check on a shutdown flag so it doesn't fight graceful shutdown.
    
*   Switch to cooperative rebalancing. One config line, no broker upgrade. Your future self will thank you.
    
*   Consider static membership for deployment survivability.
    
*   Tune your GC. Server GC, background GC, reducing Gen 2 pressure. A healthy runtime makes for a healthy health check.
    

* * *