Most of our K8s clusters use the open-source Cilium CNI, and for a subset of workloads we route outbound traffic through a dedicated Cilium Egress Gateway. The motivation is pretty common: some external systems insist on traffic originating from a known, stable IP address, so SNATing egress traffic through a designated gateway node gives us that consistency. It's a setup that's been running in production for quite some time and, for the most part, had been remarkably uneventful, and we are not complaining.
Something Broke!
That, however, changed when one of our busiest production clusters started exhibiting intermittent DNS failures. The symptoms were frustratingly inconsistent - most lookups completed successfully, but every so often an application would fail to resolve an external hostname, only to succeed moments later. Unfortunately the failures were frequent enough to cause real production impact, yet sporadic enough they escaped a very thorough, focused investigation.
DNS ?
Our first instinct was to blame DNS as It was the most obvious suspect. We started with the usual suspects:
CoreDNS
upstream resolvers
network latency
packet loss
firewall rules
Essentially everythinh you would normally expect to investigate when name resolution starts misbehaving, but nothing stood out - everything just looked reassuringly boring. Yet the application errors kept trickling in time to time, and of course on our busiest PROD cluster!
Observability To The Rescue
As the DNS investigation was a dead-end we naturally shifted our attention away to look elsewhere. Next up was the layer underneath - the Cilium CNI as it happened to be sitting squarely in the middle of every outbound packet.
As I worked my way through the dashboards, one metric immediately stood out: cilium_ct_any4_global - steadily climbing until it was brushing up against its configured limit with the timestamp matching the outages. I personally - out of my ignorance/lack of knowledge - had always associated connection tracking with TCP, so I was a bit surprised to discover through the docs. that It tracked protocols like UDP and ICMP. This was interesting, and I though to myself - If UDP is stateless, why would it need a conntrack table?
UDP Is Stateless, But ...
To be fair, that assumption wasn't entirely wrong as UDP is INDEED a stateless protocol. Unlike TCP, there is no three-way handshake etc., and the sender transmits it and moves on, with no guarantee that it will ever arrive or that anyone will respond. So seeing a connection tracking table steadily filling up with what was almost entirely DNS traffic felt fundamentally at odds with everything I thought I knew about how UDP worked.
SNAT forces the network device to remember things.
Further research cleared out few things - every outbound connection matching our egress policy was being SNAT'ed through the Cilium Egress Gateway so that external systems would always see the same source IP. The moment a gw replaces a pod's source addr with its own, it takes on a new responsibility: when the reply eventually comes back, it somehow has to figure out which pod inside the cluster originally sent the packet. And unlike TCP, he gw has no way of knowing when it's safe to forget about that translation.
Connection Tracking
Every time the gw rewrites a DNS query, it creates a small piece of state that says, "I changed this packet's source address. If a reply comes back matching these details, send it back to this pod." The reply usually arrives a few milliseconds later, the entry is used once, and after a timeout it disappears.
Packet Flow
───────────────────────────────────────
Pod ─────► Gateway ─────► DNS
│
│
▼
Create CT Entry
│
▼
DNS ◄───── Gateway ◄───── Reply
│
▼
Delete CT Entry Or After A Timeout
The problem, of course, is that production systems rarely do anything just once.
Modern applications are surprisingly chatty when it comes to DNS. A single service call can trigger one or more DNS queries. Now multiply that across hundreds of pods, each generating hundreds of lookups over relatively short periods, and suddenly the gateway isn't remembering a handful of translations anymor as it's trying to remember hundreds of thousands of them simultaneously. So the realization:
The DNS infrastructure was never the bottleneck. The gateway's memory was.
UDP hadn't magically become stateful - The gateway did.
In other words, the conntrack table wasn't simply tracking TCP connections at al, but it was tracking the gw's own state. Every time the gateway rewrote a packet, regardless of whether the protocol was TCP or UDP, it had to remember enough information to undo that rewrite when the reply returned.
Long Term Fix
There really wasn't a need for us t o track DNS traffic per customer egress IP. So in a nutshell we had effectively turned our busiest source of short-lived traffic into unnecessary work for the gateway.
So the solution was simply to stop doing that.
By excluding DNS traffic from our CiliumEgressGatewayPolicy, DNS queries bypassed the gateway entirely and exited through the node's normal networking stack instead thereby ensuring that the gw no longer had to allocate conn. tracking entries for every DNS lookup, while the application traffic that genuinely required a stable egress IP continued to flow through the gateway exactly as before.
The effect was almost immediate. The number of UDP entries in cilium_ct_any4_global dropped dramatically. Hubble showed DNS traffic leaving with the node's local IP while HTTPS traffic continued to use the gateway IP. More importantly, the intermittent DNS failures disappeared, and so did the BPF map exhaustion warnings that had quietly been building in the background.
The protocol is stateless. The infrastructure around it probably isn't.
Looking back, the most valuable lesson wasn't about Cilium, eBPF or even connection tracking. It was about abstractions.
We spend years learning that UDP is stateless, and that's absolutely true—from the protocol's perspective. But protocols don't exist in isolation. The moment you ask a device to rewrite packets, enforce policy or perform NAT, somebody has to remember what changed. That state has to live somewhere, and in Cilium's case it lives inside eBPF connection tracking maps. Once I stopped thinking about cilium_ct_any4_global as "UDP connection tracking" and started thinking of it as "the gateway's short-term memory," everything else fell neatly into place.
