# From the Vault: Designing a Hybrid Cloud Render Farm Under Time Constraints
A few years back, our on-prem. render farm ran short of capacity at the worst possible time: multiple projects were rendering simultaneously, and a large new batch of shots had just landed on top of it. We needed an urgent way to spill overflow rendering into another datacenter, and obviously the cloud emerged as the obvious choice, without blowing past the deadline.
An honest admission first up - we had no real prior experience running workloads in the cloud, so it was a genuine step into the unknown, and it was going to be hard. But the mindset going in was that we *had* to make it work, and that whatever we learned along the way would outlast the immediate deadline, and that turned out to be true.
I'll say upfront that the architecture that follows isn't perfect, and has rough edges. We had about two months to land a workable solution, so there are rough seams in here we had no time ot work over.
## Rendering Is Bursty, and Bursty Is Hard
Render farms have a shape that's familiar to anyone who's run capacity for a spiky workload: baseline demand most of the year, and punishing peaks before a deadline. And here is a rough walk-through of the proposed shape of the solution:
* Artists submit render jobs on-prem.
* Scheduler decomposes each job into independent per-frame tasks.
* During peak load, compute bursts into AWS.
* Most cloud workers run on **Spot instances** to control cost.
* Assets live in S3.
* [**FSx for Lustre**](https://aws.amazon.com/fsx/lustre/) sits in front of s3 as a shared, high-throughput fs so hundreds of workers aren't all hammering S3 for the same textures at once.
* Finished frames go back to S3, and a sync service pulls them home to the studio.
## Why Rendering Is One of the Best Workloads for Spot
Spot instances are cheap because AWS can reclaim them with few minutes' notice. That might be a dealbreaker for a lot of workloads - but rendering is close to the ideal case for it, and for three reasons:
1. **Embarrassingly parallel.** Each frame renders independently.
2. **Stateless.** A worker holds no state that matters beyond the frame it's currently on.
3. **Idempotent.** If a frame gets interrupted mid-render, the worst case is that it gets re-rendered.
Workers poll the instance metadata endpoint (`169.254.169.254`) at a fixed cadence for interruption notices. On a warning, a worker *attempts* to
* stop requesting new frames
* uploads whatever it has to the persistent storage, and
* finishes the one it's on if time allows, and
* terminates cleanly if possible
A small baseline of On-Demand ec2 instances keeps the farm going.
> The fleet itself was built with CloudFormation and Auto Scaling Groups, using **Packer-built golden AMIs** - bake as much as possible into the image, let `cloud-init` handle registration, and workers are render-ready within minutes. Fast, and easy to reason about - which is exactly what you want for something that scales up and down constantly.
## The Scheduler: Where the First Iteration Fell Apart
The scheduler is the control plane: job submission, batch decomposition, frame assignment, retries, completion tracking.
Workers are the data plane: they render and push to S3. Separating these cleanly matters, because it means the scheduler can hiccup for a few seconds without a single worker noticing.
But the first version of this design had a problem that's easy to miss until someone points it out: **making the scheduler highly available doesn't help if all the state lives in the scheduler's memory.** All you've done is move the single point of failure somewhere less visible.
The fix was to push state into the database and treat the scheduler as disposable:
* **PostgreSQL on RDS Multi-AZ** became the source of truth.
* Jobs, Frames, and Workers as first-class tables.
* Frames move through an explicit lifecycle: `PENDING → RUNNING → COMPLETED / FAILED → PERMANENTLY_FAILED`.
Once frame state lives in the database instead of in a process's memory, the scheduler itself becomes stateless and replaceable - which is what actually makes HA possible.
### The Queue Bottleneck Nobody Notices at Small Scale
The way our scheduler was handing out frames:
```sql
SELECT * FROM frames WHERE status = 'PENDING' LIMIT 1 FOR UPDATE;
```
This works fine with ten workers. With hundreds, it becomes a lock-contention nightmare - workers queue up behind each other just to claim a single row, and the scheduler's effective throughput craters exactly when you need it most.
The fix is `FOR UPDATE SKIP LOCKED` (available since Postgres 9.5), combined with **batched claims** - a worker asks for 5–10 frames at a time instead of one, which cuts the number of round-trips by an order of magnitude. Beyond a certain fleet size, even this stops being enough, and the honest answer is to move the queue to something built for it - Redis, SQS, or Kafka. Postgres-as-a-queue is a fine choice until it isn't - but we never got to re-implement this portion.
### Leader Election Has a Split-Brain Problem Too
Multiple scheduler instances need to agree on who's actually in charge, which is a classic use case for a **Postgres advisory lock**. But leader election alone doesn't prevent split-brain: if a leader stalls (GC pause, network blip) and a new one is elected, the *old* leader can wake up and keep assigning frames, unaware it's been replaced.
The fix is **epoch fencing**. Every elected leader gets a monotonically increasing epoch number, and every frame assignment carries that epoch. A stale leader with an old epoch simply can't overwrite assignments made by a newer one. It's a small mechanism, but it's the difference between "we have leader election" and "we have *correct* leader election."
## The Control Loops We Missed
Initially we only (and embarrassingly) only designed the "happy-path" - the case where everything succeeds, and learnt it the hard way that a real system needs to answer: *what happens when it doesn't?*
**The Frame Reaper.** A worker can die mid-frame - Spot reclamation, a kernel panic, a network partition - and if nothing is watching, that frame sits in `RUNNING` forever, invisibly stuck. A reaper loop runs every 30 seconds, checks worker heartbeats, and flips stale `RUNNING` frames back to `PENDING` with an incremented retry count.
**Poison frame handling.** Some frames are just bad - a corrupt texture, a scene file that crashes the renderer. Without a cap, a poison frame retries forever and quietly eats capacity. `max_retries = 5`, then `PERMANENTLY_FAILED` and a notification to the artist who submitted it.
**Priority that's actually enforced.** Priority existed early on as a column with no teeth - nothing in the scheduler actually respected it. Fixing that meant an explicit `ORDER BY priority` in the claim query, plus a reserved fraction of workers for background jobs so low-priority work doesn't starve indefinitely.
None of these are exciting to design. All three are the difference between a system that survives a bad week and one that needs a human babysitting it.
## Storage: Know What Your Source of Truth Actually Is
S3 is durable storage. **FSx for Lustre is not** - it's a performance cache sitting in front of S3, and conflating the two is a fast way to lose data. Workers writing to `/mnt/lustre/fs/job/seq/scn/assets//` does *not* mean that data exists in S3 unless it's explicitly exported. The production-safe pattern: workers write outputs **directly to S3**, and Lustre is used almost exclusively for reading input assets, avoiding the failure mode where 200 workers each try to pull the same 30 GB asset set independently.
**Prewarming** followed the same "define it operationally or it's not real" pattern. "Prewarm the cache" isn't a plan - it became: the scheduler builds an asset manifest for a job, a dedicated prewarm worker walks that manifest, and simply *reading* each file forces Lustre to pull it from S3 ahead of time. Job state gained an explicit `PREWARMING` stage so the system's state machine matches what's actually happening physically.
## Worker Identity: Simpler Beat Fancier
The first pass at worker authentication was mTLS - which sounds appropriately serious for a distributed fleet, until you have to answer "how do we issue, rotate, and revoke certificates for autoscaling workers that live for twenty minutes?" That's a real operational burden for a workload that doesn't need it.
The simpler and more realistic answer: workers authenticate using their **EC2 Instance Identity Document**, which the scheduler validates against AWS's signature, with TLS handling transport security. It's a good example of a broader principle - the "more secure-looking" option isn't automatically the right one if it introduces lifecycle management you don't actually need.
## Hybrid Networking and the Cost of Forgetting Bandwidth
The studio connects to AWS over Direct Connect with VPN as a fallback, and results flow AWS → S3 → a studio-side sync service (workers never write directly to studio storage). The detail that's easy to miss: a large enough result-sync job can **saturate Direct Connect** and start starving control-plane traffic - the very traffic your scheduler and workers need to keep functioning. The fix is unglamorous but necessary: bandwidth limits and bulk sync windows, with control-plane traffic always prioritized.
## Autoscaling, Boot Storms, and Why 99.9% Doesn't Compose
At fleet scale, synchronized behavior becomes its own failure mode - hundreds of workers booting at once and hammering the same registration endpoint or polling loop at the same moment. Random startup jitter and frame batching smooth this out. Scaling itself needs hysteresis and cooldowns, or you end up oscillating capacity up and down in response to noise.
The broader lesson, and probably the most transferable one in the whole exercise, is about availability math. Five services each running at 99.9% don't combine to 99.9% - they compose multiplicatively:
```plaintext
0.999^5 ≈ 99.5%
```
Every synchronous dependency you add is a tax on your overall availability, and microservice architectures pay this tax constantly through added network hops, retry storms, and fan-out amplification. Circuit breakers exist specifically to stop retry storms from compounding a small failure into a large one - instead of hammering a struggling downstream with retries, the circuit opens and requests fail fast, protecting the thing that's already in trouble.
It's also, frankly, a solid argument for **modular monoliths** as a starting point: clear module boundaries, no network calls between them, simpler deployments, and - critically - higher baseline availability, because you haven't yet introduced the dependencies that erode it. Split into microservices when there's a real reason to, not by default.
## What This Exercise Was Actually Teaching
Strip away the render-farm specifics and a handful of principles remain, and I think they generalize to basically any distributed system:
1. **Separate control plane from data plane.** If task state is durable, the scheduler can fail for a few seconds and nothing downstream notices.
2. **Know your source of truth, explicitly.** S3 was durable; FSx was a cache. Confusing a performance layer for a durability layer is how data quietly disappears.
3. **Every control loop matters.** The happy path is not a design - it's a slide. Reapers, retry limits, poison-record handling, and bandwidth governors are what make a system survive contact with reality.
4. **Distributed systems are a series of tradeoffs, not a series of best practices.** Performance vs. resilience, cost vs. availability, simplicity vs. scale - the right answer changes with fleet size, and a good design says *when* it changes, not just *what* the current answer is.
5. **Availability is architectural, not aspirational.** You don't get high availability by choosing highly-available components. You get it by reducing synchronous dependencies, placing redundancy where it actually matters, and designing explicitly for degradation instead of hoping it never happens.
What started as "how do I phrase a resume bullet about a hybrid render farm" turned into something closer to a staff-level design review - not because the technology got more exotic, but because the questions got sharper. Every fix above came from someone asking *"okay, but what happens when this fails?"* one level deeper than the previous draft had gone. That's really the whole job.