Discussion on "System Design Drill Day 5: Load Balancing and Horizontal Scaling"

Anish Ansari · 2026-10-23T03:13:34.244Z

You add a second app server before a big sale. Users start getting logged out at random, product images show up for some people and not others, and the daily report email arrives twice. The code didn'

One check before the third server goes live: add up the database connections each box will open. Three servers with their usual worker pools can hit MySQL's limit together, even when traffic looks fine.

Right, once the check name is in the log line nobody has to dig for it. It also makes patterns jump out when the same step keeps stalling across runs.

That pattern point is worth acting on. Counting failures by check name would show whether one step stalls repeatedly or many steps stall once, and each points to a different fix.

A small weekly check on that count would close the loop: if one check keeps topping the list, that is the step to fix first, rather than treating every failure as a one-off.

The rolling-deploy exercise would be even stronger with a checkout running while old and new application versions overlap. Database compatibility is one boundary; session payloads and queued-job formats are two more. Testing that mixed-version window makes “any server can handle any request” a property of the release process as well as the steady state.

Agreed Mahdi, the article only covers backward-compatible migrations, but session payloads and queued jobs cross that old and new window too. Running a real checkout through a half-rolled deploy is a nice way to prove it.

One gotcha I'd add from a rollout like this: when you sync APP_KEY across the new boxes, every existing encrypted cookie dies at the same moment. We watched a chunk of users get bounced mid session during the cutover. If you're on Laravel 11, APP_PREVIOUS_KEYS lets you rotate instead of flipping the switch, so old cookies keep decrypting until they expire naturally. Worth calling out before the big sale, not after.

Good catch Kamran, a hard APP_KEY swap logging everyone out in the middle of a sale is a painful way to learn that. Rotating with APP_PREVIOUS_KEYS so old cookies keep decrypting is the much safer move.

One crash fixture I would add: a worker sends the report email, then dies before recording completion. A scheduler lock stops concurrent starts, but a retry can still repeat that external side effect. Keep the same report/run ID on retry and test the provider's duplicate handling, where supported. Where it cannot deduplicate, record that remaining risk. Would you include that crash point in the Day 3/Day 5 exercise?

Yes Arnold, that crash point is worth adding, since the scheduler lock only stops two runs starting together and does nothing about a retry repeating the send. Reusing the same run ID and checking it before sending covers most of it, and where the provider can't dedupe you just note the leftover risk.

The readiness-vs-liveness split is the detail most first-time setups get wrong: folding Redis into readiness means one cache blip pulls every server out of rotation at once. The duplicate report email is the other classic: a scheduled job needs a single owner the moment more than one box exists.

Thanks Abhinav, keeping Redis out of readiness is exactly what stops one cache blip from emptying the whole pool. And the duplicate report email is the bug that only shows up the day you add a second server 🙂

Search Hashnode

System Design Drill Day 5: Load Balancing and Horizontal Scaling

Responses54