While managing our internal K8s platform I've come across multiple instances where the onboarding team(s) haven't really figured out the workload resource allocation math correctly thereby leading to unexpected results.
For instance when we deploy an app. to K8s, we might think we have the resource calculations perfectly figured out. If we have a 2Gi memory quota in our ns, and the deployment needs 1.8Gi of memory, everything should work smoothly, right? Unfortunately and possibly no as K8s has a few hidden math traps that frequently break deployments and cause pods to get stuck. Here is exactly how K8s handles memory limits and scheduling behind the scenes.
The Rolling Update Issue
The very first mistake people make is looking at a deployment's steady state. If you multiply your pod's memory limits by the number of replicas, you get your normal footprint. However, K8s rarely runs exactly that number of replicas during an update. By default, deployments use a rolling update strategy with a setting called maxSurge. This means it would temporarily spins up extra new pods before killing the old ones to prevent downtime. If your resource quota is tuned exactly to your normal running capacity, the rolling update will instantly hit the limit wall, and your new pods will be blocked. YOu might have to stream the ns specific events to figure this one out -
kubectl get events --sort-by=.lastTimestamp
Init Containers Twist the Math
Things could also get more interesting when you add init containers into the mix. Because init containers run Serially and finish before the main app. starts, they never run at the same time as your app. Because of this, K8s calculates your pod’s memory requirements using an "effective request" formula. The scheduler compares the sum of all your regular app containers against the single highest resource request among your init containers. Whichever number is bigger becomes the official memory requirement that the scheduler uses to find a node.
The Init Container Excessive Reservation Problem
This scheduling logic creates a strange side effect that might be called a "ghost reservation." If you have a heavy init container that needs 1Gi of memory to run a quick database migration, and an app container that only needs 100Mi, the scheduler permanently logs that pod as needing 1Gi of memory. Even after the init container finishes and hence is not using that memory, the scheduler does not give that space back to the node. That 1Gi remains blocked off for the entire lifecycle of the pod, which can easily cause your cluster nodes to look fuller when they are actually sitting idle. Take a look at this discussion for more information.
Memory Backed volumes count towards Memory Limit
The memory limit for the Pod or container can also apply to pages in memory backed volumes, such as an emptyDir. The kubelet tracks tmpfs emptyDir volumes as container memory use, rather than as local ephemeral storage.
Only came across this once one of our clients suffered a major outage unintentionally due to the fact (more to follow later in a separate blog post).
As medium: Memory leverages tmpfs (RAM) under the hood, writing to that directory allocates pages in the the page cache whose usage is is tracked by the kubelet under the container memory cgroup, it might create a series of catastrophic blind spot.
Limits vs Requests for the Scheduler
This often has been a source of constant confusion on our platform. yet another piece of puzzle is understanding what the scheduler actually cares about. The kube-scheduler completely ignores your memory limits when deciding where to place a pod; it only looks at memory requests.
The scheduler ensures that, for each resource type, the sum of the resource requests of the scheduled containers is less than the capacity of the node.
However, if you specify a memory limit for your container but forget to define a request, k8s automatically sets the request equla to your limit. In that specific scenario, your limit indirectly controls your scheduling, which can accidentally trigger more reservation than actually being used leading to wasted resources due to reduced allocatable capacity, or push your deployment past your ns hard resource quota silently.
Silent Danger of CPU Throttling
Another issue we've observed sneaking up silently on dev. teams is this one. While breaking a memory quota results in an immediate, loud OOMKilled error, misconfiguring your CPU limits creates a much more frustrating, silent failure.
CPU is a compressible resource.
When your app. hits its CPU limit, K8s doesn't kill the pod; it throttles it by slowing down its processing time. This causes your application to become incredibly slow and unresponsive. Because the app stops responding in time, its own livenessProbe or readinessProbe health checks will begin to fail. K8s will then assume the container is dead and trigger a cascading loop of unneeded restarts, all because a strict CPU quota starved the application of the processing power it needed to answer its own health checks.
