# How Invisible Default Setting Can Break Production- War Story ```plaintext tail -f /var/log/cassandra/system.log ... INFO [main] 2026-06-21 17:44:42 Cassandra starting up... WARN [Thread-1] 2026-06-21 17:44:43 High memory usage detected ERROR [Thread-2] 2026-06-21 17:44:44.408 java.lang.OutOfMemoryError: unable to create new native thread ``` As a senior engineer, you get used to things breaking. But sometimes, the root cause is hiding a few layers deep in the infrastructure. Recently, our company messenger app went down completely. I was looped into the incident bridge to figure it out. Here is a quick write-up of the investigation, how we traced it from K8s all the way down to a container runtime default, and what we learned. ### The Outage: Starting at the Top Our app stack is mostly running on a K8s platform. Naturally, that is the first place I looked when the alerts started firing. I opened the logs for the main messenger workload. Immediately, I saw that the application was throwing `503 Service Unavailable` errors when trying to call one of its downstream services. In our architecture, the communication to this downstream service goes through an HAProxy load balancer layer. Some people always want to blame the load balancer first. To verify, we checked the HAProxy logs. HAProxy was showing connectivity issues to the backend with `504 Gateway Timeout`. This actually ruled HAProxy out. It was doing its job, trying to pass traffic, but the backend was not responding in time. ### Moving to the Downstream Service Next, we checked the downstream backend service in K8s. The pods for this service were in a `CrashLoopBackOff` state. We looked at the logs right before the crash. We figured out that the service was trying to talk to our Cassandra database cluster, but the response latency had increased massively. The BE app. was basically timing out waiting for the database, exhausting its connections, and crashing. So, the backend was just a victim. The real problem was in the database layer. ### The Database Layer: The Real Culprit We moved our investigation to the Cassandra nodes. For this setup, Cassandra is not running inside K8s. It is running on instances using Podman containers. When we checked the cluster status, we saw that 3 to 5 nodes had been OOM'ing (Out of Memory) since the morning. They were constantly getting killed and restarted. We went into Grafana and pulled the Loki logs for the Cassandra containers. We found this exact error spamming the logs: ```text 2026-06-21 17:44:44.408 java.lang.OutOfMemoryError: unable to create new native thread ``` ### The "Aha!" Moment When you see an \`OOM Error\` in Java, the first thought is usually heap exhaustion. But the `unable to create new native thread` part is very specific. This error doesn't mean the JVM ran out of RAM. It means the operating system (or the container runtime) is refusing to let the JVM spawn any more threads. Cassandra is a very heavy Java application; it relies on creating a lot of native threads for connection pooling, compaction, and request handling. We investigated the host and the container configs. That is when we found it: **the Podman default thread limit (pids-limit).** By default, Podman sets a limit on the number of processes/threads a container can create to prevent fork bombs. During high traffic that morning, Cassandra tried to scale up its thread count to handle the load. It hit the Podman default PIDs limit limit. Because the JVM was blocked from creating the threads it needed to operate, it panicked, threw the OOM error, and crashed the node. This caused a chain reaction: 1. Cassandra nodes hit thread limit and crash. 2. Database latency spikes for surviving nodes. 3. Downstream K8s service times out waiting for DB, then crashloops. 4. HAProxy throws 504s. 5. Frontend app gets 503s and goes down. ### The Immediate Fix To resolve this, we had to update the container configuration for the Cassandra nodes to increase or remove the `pids-limit` restriction, allowing the JVM to spawn the threads it actually needed for our production scale. Once we restarted the Podman containers with the new limits, the database stabilized, the downstream service recovered, and the messenger app came back online. ### Bigger Picture: Architectural Changes Fixing the Podman thread limit was the immediate band-aid to stop the bleeding. But incident reviews are about finding long-term structural fixes. Our single massive production database had too large of a blast radius. The biggest takeaway from this outage was the decision to split our single production instance into two separate environments: Large Prod and Small Prod, each tailored to different audienses.. 1. **Moving Large Prod to k8s:** For our main, larger prod. workload, we are migrating Cassandra into our k8s platform. Yes, I know running heavy stateful databases in K8s is a controversial topic. But for us, having our database on the same control plane as our application stack brings operational consistency. It gives us better automated distribution, easier pod anti-affinity rules, and faster recovery times when a node dies, compared to managing standalone container instances. 2. **Moving Small Prod to Bare Metal** For the smaller production cluster, we are going the opposite direction: pure bare metal. Yes, maintaining bare metal adds a different layer of operational complexity because we have to manage the physical nodes directly using our automation playbooks instead of K8s manifests. However, removing the container runtime, the virtualized networking layer, and the kernel isolation overhead gives us the absolute maximum raw disk I/O and compute performance for our most latency-sensitive workloads. Sometimes, the best architectural decision is knowing when to use the shiny orchestration tool, and when to just let a database talk directly to the hardware. ### Takeaway When you are managing infrastructure, you monitor CPU and memory limits very closely. But resource limits like process/thread counts (PIDs) are easy to forget until they break production. Always check the default limits of your container runtime (Docker, containerd, Podman) because their defaults are usually meant for generic lightweight web apps, not heavy enterprise databases.