As k8s engineers, we're constantly looking for ways to make our platforms faster, scalable, reliable, and more efficient.
We tune resource requests, optimize autoscaling, benchmark storage backends, and chase n/w bottlenecks. Yet over the course of managing various platforms over the years I've noticed that the most impactful performance wins rarely come from simply adding more CPU, mem. or storage - they come from reducing the amount of work the OS has to do in the first place.
Few days back chasing a production bug sent me down an interesting rabbit hole. It started with investigation that lead to a Linux syscall called sendfile, took me through zero-copy networking and DMA, and eventually helped me understand why eBPF, XDP, and Cilium have become such important building blocks for modern k8s platforms.
These technologies solve different problems, but they share one philosophy:
The fastest operation is the one you never have to perform.
A Pattern Hidden Throughout Linux
When we hit a performance issue, our instinct is to ask:
How can I make this operation faster?
Kernel developers tend to ask something different:
Can I avoid doing this operation altogether?
That mindset shows up everywhere in the kernel. Instead of copying data, traversing networking layers, context switching between kernel and userspace, or eval'ing thousands of rules per packet, Linux keeps evolving toward doing less.
Once you see the pattern, technologies like sendfile(), eBPF, and Cilium stop looking like unrelated tools and start looking like the same idea applied in different places.
The Cost of Moving Data
Consider a traditional app serving a file over the network. A normal implementation that I've comre across most generally -
read(file_fd, buffer, size);
write(socket_fd, buffer, size);
The data path looks roughly like:
Disk
↓ (DMA copy)
Kernel Page Cache
↓ (CPU copy)
Userspace Buffer
↓ (CPU copy)
Kernel Socket Buffer
↓ (DMA copy)
NIC
That's 4 copies of the same bytes before they leave the machine — two of them burning CPU and polluting its caches - plus 4 context switches, since read() and write() each cross into the kernel and back.
Each copy costs CPU cycles, memory bandwidth, and cache capacity. At small scale it's negligible. At large scale it's expensive.
How sendfile() Changes the Equation
Linux added sendfile() to cut out the copies that route through userspace. Instead of read() then write(), the app calls:
sendfile(socket_fd, file_fd, NULL, size);
In the best case the path collapses to:
Disk
↓ (DMA copy)
Kernel Page Cache
↓ (DMA gather copy → refs., no cpu copy)
NIC
The app never touches the payload, and we drop from four context switches to two.
One honest caveat worth keeping: this true zero-copy path only happens when the NIC supports scatter-gather DMA and checksum offload. There, the kernel hands the NIC references to the page-cache pages and the hardware gathers them directly. Without that hardware support, sendfile() still saves you the two userspace copies, but there's one remaining kernel-side copy from the page cache into the socket buffer. So sendfile() is "fewer copies" always, and "zero-copy" when the hardware cooperates.
Either way, the lesson isn't the syscall. It's the design principle behind it:
Avoid moving data across boundaries unless the data has changed.
DMA: The Foundation Underneath
That zero-copy path doesn't work by magic. It leans on something much older than sendfile() — DMA (Direct Memory Access).
DMA predates all of this by decades, and it's precisely why the page-cache-to-NIC hop above can skip the CPU. Modern hardware can move data directly between devices and memory without the CPU touching every byte:
NVMe SSD ↔ RAM
NIC ↔ RAM
GPU ↔ RAM
The CPU's job shrinks to:
Configure the transfer
Initiate it
Get out of the way
sendfile() is really just the kernel arranging for DMA to do the heavy lifting. Same theme, one layer down:
Reduce unnecessary CPU involvement.
The Same Problem Appears in k8s Networking
Take a packet entering a cluster that uses iptables-based svc routing. A simplified path:
NIC
↓
Kernel Networking Stack
↓
Conntrack
↓
iptables Rules (NAT + forwarding)
↓
Pod
Every layer does work. Every lookup costs cycles. Every packet traverses multiple subsystems before it reaches its destination.
At moderate scale this is fine. At large scale — service meshes, API gateways, observability pipelines, high-throughput microservices, AI inference workloads — the networking stack itself becomes a meaningful CPU consumer.
The Traditional kube-proxy Challenge
Historically, k8s svc load balancing leaned heavily on iptables. iptables is wonderfully flexible, but it wasn't designed for cloud-native environments slinging massive east-west traffic at scale.
The packet inspection work is done by the kernel's netfilter. And The real problem is how those rules are evaluated. In iptables mode, rules are a sequential list, so match time scales roughly linearly (O(n)) with the number of svc and endpoints. For every packet the kernel may walk through:
Packet
↓
Rule Matching (linear scan)
↓
NAT Processing
↓
Conntrack Lookup
↓
Forwarding Decision
As svc count grows, so does the chain you scan per packet. This is exactly why kube-proxy later gained an IPVS mode, which uses hash-table lookups (closer to O(1)) instead of a linear scan — a meaningful improvement that predates eBPF entirely.
So to conclude iptables was traditionally just not designed for such large volume of work. It's that, for this workload, the system was doing more work than necessary.
eBPF: Bringing Logic Closer to the Data
This is where eBPF gets interesting.
Worth saying up front: eBPF is much bigger than networking. It powers tracing, observability, and security (seccomp-BPF, LSM hooks, and more). But in this story, the relevant trick is networking.
eBPF lets you run sandboxed programs directly inside the kernel. Instead of pushing packets through layer after layer, your forwarding logic can run much earlier in the packet's journey.
Traditional path:
Packet
↓
Networking Stack
↓
iptables (linear scan)
↓
Additional Processing
↓
Destination
eBPF path:
Packet
↓
eBPF Program
↓
Destination
The payload is identical. What's reduced is the work needed to make a forwarding decision.
Why Cilium (and other eBPF based solutions) Matters
When I first started working with Cilium, I realized it embodies the same principles that motivated sendfile().
Cilium uses eBPF to move networking decisions closer to where packets enter the kernel. Instead of relying primarily on large iptables rule sets, svc handling can run as efficient kernel-resident eBPF programs. (In full kube-proxy replacement mode it removes the iptables-based svc path entirely; some configurations still keep iptables around for parts of the stack.)
The result is often:
The goal isn't to make packet processing faster. It's to do less packet processing.
A Common Thread Across Linux Innovation
Step back and a lot of Linux performance work follows one pattern.
sendfile()
Avoid unnecessary data copies.
DMA
Avoid unnecessary CPU involvement. (And it's the foundation sendfile() stands on.)
io_uring
Avoid unnecessary syscall overhead, via shared submission/completion rings instead of a syscall per op.
eBPF
Avoid unnecessary networking-stack traversal.
XDP
Avoid unnecessary processing by running at the driver level, before the kernel even allocates an sk_buff.
Cilium
Avoid unnecessary svc-routing overhead inside k8s.
Different subsystems. Same philosophy.
What This Changed for Me as a k8s Engineer
Studying sendfile(), DMA, eBPF, and Cilium changed how I think about performance. Now when I chase a bottleneck, I ask:
How many times is the data being copied?
How many components are touching this packet?
How many layers are involved in this request?
Can some of this work be eliminated entirely?
Those questions surface opportunities that CPU graphs and memory dashboards never might.
Final Thoughts
The most valuable thing I took from this wasn't a syscall or a CNI. It was a recurring pattern in Linux design: the kernel community keeps finding ways to eliminate work.
Less copying. Less context switching. Less packet processing. Less CPU involvement.
Whether it's sendfile(), DMA, io_uring, eBPF, XDP, the principle holds:
The best optimization is often removing work rather than accelerating it.
As k8s engineers, that mindset is worth carrying into our own platforms. Next time you're debugging a performance issue, don't only ask:
How can I make this faster?
Ask:
What work can I avoid doing altogether?
