# The Fastest Operation Is the One You Never Perform As k8s engineers, we're constantly looking for ways to make our platforms faster, scalable, reliable, and more efficient. We tune resource requests, optimize autoscaling, benchmark storage backends, and chase n/w bottlenecks. Yet over the course of managing various platforms over the years I've noticed that the most impactful performance wins rarely come from simply adding more CPU, mem. or storage - they come from reducing the amount of work the OS has to do in the first place. Few days back chasing a production bug sent me down an interesting rabbit hole. It started with investigation that lead to a Linux syscall called [`sendfile`](https://man7.org/linux/man-pages/man2/sendfile.2.html), took me through zero-copy networking and DMA, and eventually helped me understand why [eBPF](https://ebpf.io/what-is-ebpf/), [XDP](https://en.wikipedia.org/wiki/Express_Data_Path), and [Cilium](https://en.wikipedia.org/wiki/Cilium_\(computing\)) have become such important building blocks for modern k8s platforms. These technologies solve different problems, but they share one philosophy: > The fastest operation is the one you never have to perform. * * * ## A Pattern Hidden Throughout Linux When we hit a performance issue, our instinct is to ask: > How can I make this operation faster? Kernel developers tend to ask something different: > Can I avoid doing this operation altogether? That mindset shows up everywhere in the kernel. Instead of copying data, traversing networking layers, context switching between kernel and userspace, or eval'ing thousands of rules per packet, Linux keeps evolving toward doing *less*. Once you see the pattern, technologies like `sendfile()`, eBPF, and Cilium stop looking like unrelated tools and start looking like the same idea applied in different places. * * * ## The Cost of Moving Data ![](https://cdn.hashnode.com/uploads/covers/6a25ff63980616b1c020fc7a/833c444f-42fb-4355-80d0-34786c71cc14.svg align="center") Consider a traditional app serving a file over the network. A normal implementation that I've comre across most generally - ```c read(file_fd, buffer, size); write(socket_fd, buffer, size); ``` The data path looks roughly like: ```plaintext Disk ↓ (DMA copy) Kernel Page Cache ↓ (CPU copy) Userspace Buffer ↓ (CPU copy) Kernel Socket Buffer ↓ (DMA copy) NIC ``` That's 4 copies of the same bytes before they leave the machine — two of them burning CPU and polluting its caches - plus 4 context switches, since `read()` and `write()` each cross into the kernel and back. Each copy costs CPU cycles, memory bandwidth, and cache capacity. At small scale it's negligible. At large scale it's **expensive**. * * * ## How `sendfile()` Changes the Equation Linux added `sendfile()` to cut out the copies that route through userspace. Instead of `read()` then `write()`, the app calls: ```c sendfile(socket_fd, file_fd, NULL, size); ``` In the best case the path collapses to: ```text Disk ↓ (DMA copy) Kernel Page Cache ↓ (DMA gather copy → refs., no cpu copy) NIC ``` The app never touches the payload, and we drop from four context switches to two. One honest caveat worth keeping: this *true* zero-copy path only happens when the NIC supports scatter-gather DMA and checksum offload. There, the kernel hands the NIC references to the page-cache pages and the hardware gathers them directly. Without that hardware support, `sendfile()` still saves you the two userspace copies, but there's one remaining kernel-side copy from the page cache into the socket buffer. So `sendfile()` is "fewer copies" always, and "zero-copy" when the hardware cooperates. Either way, the lesson isn't the syscall. It's the design principle behind it: > Avoid moving data across boundaries *unless* the data has changed. * * * ## DMA: The Foundation Underneath That zero-copy path doesn't work by magic. It leans on something much older than `sendfile()` — [DMA](https://en.wikipedia.org/wiki/Direct_memory_access) (Direct Memory Access). DMA predates all of this by decades, and it's precisely *why* the page-cache-to-NIC hop above can skip the CPU. Modern hardware can move data directly between devices and memory without the CPU touching every byte: ```text NVMe SSD ↔ RAM NIC ↔ RAM GPU ↔ RAM ``` The CPU's job shrinks to: 1. Configure the transfer 2. Initiate it 3. Get out of the way `sendfile()` is really just the kernel arranging for DMA to do the heavy lifting. Same theme, one layer down: > Reduce unnecessary CPU involvement. * * * ## The Same Problem Appears in k8s Networking Take a packet entering a cluster that uses iptables-based svc routing. A simplified path: ```text NIC ↓ Kernel Networking Stack ↓ Conntrack ↓ iptables Rules (NAT + forwarding) ↓ Pod ``` Every layer does work. Every lookup costs cycles. Every packet traverses multiple subsystems before it reaches its destination. At moderate scale this is fine. At large scale — service meshes, API gateways, observability pipelines, high-throughput microservices, AI inference workloads — the networking stack itself becomes a meaningful CPU consumer. * * * ## The Traditional [kube-proxy](https://kubernetes.io/docs/reference/networking/virtual-ips/) Challenge Historically, k8s svc load balancing leaned heavily on iptables. [iptables](https://en.wikipedia.org/wiki/Iptables) is wonderfully flexible, but it wasn't designed for cloud-native environments slinging massive east-west traffic at scale. The packet inspection work is done by the kernel's netfilter. And The real problem is how those rules are evaluated. In iptables mode, rules are a sequential list, so match time scales roughly **linearly (O(n))** with the number of svc and endpoints. For every packet the kernel may walk through: ```text Packet ↓ Rule Matching (linear scan) ↓ NAT Processing ↓ Conntrack Lookup ↓ Forwarding Decision ``` As svc count grows, so does the chain you scan per packet. This is exactly why kube-proxy later gained an [**IPVS**](https://kubernetes.io/docs/reference/networking/virtual-ips/) **mode**, which uses hash-table lookups (closer to O(1)) instead of a linear scan — a meaningful improvement that predates eBPF entirely. So to conclude iptables was traditionally just not designed for such large volume of work. It's that, for this workload, the system was doing more work than necessary. * * * ## eBPF: Bringing Logic Closer to the Data This is where eBPF gets interesting. Worth saying up front: eBPF is much bigger than networking. It powers tracing, observability, and security (seccomp-BPF, LSM hooks, and more). But in *this* story, the relevant trick is networking. eBPF lets you run sandboxed programs directly inside the kernel. Instead of pushing packets through layer after layer, your forwarding logic can run much earlier in the packet's journey. Traditional path: ```text Packet ↓ Networking Stack ↓ iptables (linear scan) ↓ Additional Processing ↓ Destination ``` eBPF path: ```text Packet ↓ eBPF Program ↓ Destination ``` The payload is identical. What's reduced is the work needed to make a forwarding decision. * * * ## Why Cilium (and other eBPF based solutions) Matters When I first started working with Cilium, I realized it embodies the same principles that motivated `sendfile()`. Cilium uses eBPF to move networking decisions closer to where packets enter the kernel. Instead of relying primarily on large iptables rule sets, svc handling can run as efficient kernel-resident eBPF programs. (In full **kube-proxy replacement** mode it removes the iptables-based svc path entirely; some configurations still keep iptables around for parts of the stack.) The result is often: * Lower latency * Reduced CPU consumption * Faster svc lookups (hash/map-based instead of linear) * Better scalability at large cluster sizes The goal isn't to make packet processing faster. It's to do *less* packet processing. * * * ## A Common Thread Across Linux Innovation Step back and a lot of Linux performance work follows one pattern. ### `sendfile()` Avoid unnecessary data copies. ### DMA Avoid unnecessary CPU involvement. (And it's the foundation `sendfile()` stands on.) ### `io_uring` Avoid unnecessary syscall overhead, via shared submission/completion rings instead of a syscall per op. ### eBPF Avoid unnecessary networking-stack traversal. ### XDP Avoid unnecessary processing by running at the driver level, *before* the kernel even allocates an `sk_buff`. ### Cilium Avoid unnecessary svc-routing overhead inside k8s. Different subsystems. Same philosophy. * * * ## What This Changed for Me as a k8s Engineer Studying `sendfile()`, DMA, eBPF, and Cilium changed how I think about performance. Now when I chase a bottleneck, I ask: ### `How many times is the data being copied?` ### `How many components are touching this packet?` ### `How many layers are involved in this request?` ### `Can some of this work be eliminated entirely?` Those questions surface opportunities that CPU graphs and memory dashboards never might. * * * ## Final Thoughts The most valuable thing I took from this wasn't a syscall or a CNI. It was a recurring pattern in Linux design: the kernel community keeps finding ways to eliminate work. Less copying. Less context switching. Less packet processing. Less CPU involvement. Whether it's `sendfile()`, DMA, `io_uring`, eBPF, XDP, the principle holds: > The best optimization is often removing work rather than accelerating it. As k8s engineers, that mindset is worth carrying into our own platforms. Next time you're debugging a performance issue, don't only ask: > How can I make this faster? Ask: > What work can I avoid doing altogether?