← all topics Client CI/CD board · say it in this order 1 Problem2 Entities + API 3 High-level design4 Deep dives 5 Final + levelsreset

Design an end-to-end CI/CD pipeline for client updates

2,000 commits a day · 10,000 server instances · 200M client installs · Hello Interview flow, board rules, one idea per line
SAY THIS SENTENCE FIRST

"There are two last miles here and only one of them is mine. I control when a server changes, so that half is a rollout problem I can solve with canaries and health gates. I do not control when a client changes, so several client versions are always live at once and I can never un-ship one. That single fact drives everything: the server deploys first and stays backward compatible, the patch is downloaded days before it is activated, and the only instant lever I have over code already on a player's machine is a server-side flag."

STEP 1 OF 5

Understanding the problem

5 min

Functional Requirements

  1. A merged commit should reach production servers automatically, through gates, without someone shepherding it.
  2. A client patch should reach players as a staged rollout, downloaded in the background and resumable.
  3. A bad release should be stoppable and its damage neutralisable fast, for both the server and the client.
  4. Clients that have not updated yet should keep working while a rollout is in flight.

Below the line (out of scope):

  • The engine and asset build system itself. I care that it produces a deterministic artifact, not how it compiles shaders.
  • Cheat detection, telemetry schemas, and the localisation pipeline. They ride the same rails and would each eat the hour.
  • Developer environment tooling and local iteration. Real, and a different design.
  • Store review stays IN scope as a constraint, not as a workflow. It takes hours to days and it is the reason risky behaviour ships as configuration rather than as a binary. Cutting it entirely would remove the most interesting force on the design.

Non-Functional Requirements

Do the bandwidth math first, because it shows where the simple design breaks
  • One 80 MB patch to 50M players is 4 PB of egress, more than the release hour can carry.
    • 50M daily players times an 80 MB patch is 4 PB of egress for one release.
    • Delivered in a single hour that is 1.1 TB a second, which is not a capacity anyone can buy on demand.
    • Spread over 48 hours it is 23 GB a second, which is an ordinary CDN commitment.
    • So delivery has to be spread over time, which makes it a bandwidth requirement before it is a safety one.
    • It also costs real money, and the counterfactual is the number that lands: at roughly two cents a gigabyte, 4 PB of deltas is about 80,000 dollars, while shipping the full 4 GB build to the same 50M players would be 200 PB and about 4 million. That gap is why delta patching is a budget line and not an optimisation. Across all 200M installs rather than the daily actives, scale both by four.
  1. Support the current client and the two versions before it (N-2) for at least 30 days, because players update when they feel like it.
    • Players update when they feel like it, so at any moment three or more client versions are live, for weeks.
    • Support N-2 for at least 30 days. That is a real tax: every protocol change is carried three times over.
  2. PR feedback takes under 10 minutes, and a merge reaches the canary in under an hour. 500 engineers and 2,000 commits a day means a slow pipeline is not an inconvenience, it is a queue that never drains.
  3. A client patch downloads in the background, resumes after a drop, and never blocks launch unless the protocol genuinely broke. A forced full-stop update is the worst thing you can do to a player who has 20 minutes to play.
  4. A rollout must be haltable globally within about 60 seconds, and a bad code path switchable off without any download at all.
  5. Server deploys must not drop live sessions. A match is stateful, so a server cannot simply restart under it.
  6. Every artifact is immutable and content-addressed. What was tested must be bit-identical to what ships, which makes promotion a pointer move rather than a rebuild.
DONE WHEN: the interviewer has heard "4 PB, so staging is a bandwidth requirement" and "I cannot un-ship a client". Those two license the manifest design and the flag layer.
STEP 2 OF 5

Entities + API

5 min

Defining the Core Entities

  • Artifact is one built output addressed by the hash of its contents: a server image, a client executable, a pack file. Immutable by construction, because the name IS the content.
  • Release bundles the artifacts that belong together: a server image, a client bundle, and the manifest that describes it. One version number spans all three, which is what makes "are these compatible" answerable.
  • Manifest is the answer to "what should THIS install be running", given platform, channel, cohort and current version.
    • It is the most important entity and the one people forget.
    • It lists content-addressed chunks, so the client downloads only what it does not already have.
    • Flipping the manifest is the release. Everything before that is just moving bytes around.
  • Channel is the audience track: internal dogfood, alpha, beta, live. It is the cheapest safety mechanism available and it is the stage BEFORE the 1 percent cohort, so a release has usually been played by employees for a week before a stranger sees it.
  • Cohort is a stable bucket of installs within a channel, derived by hashing the install id. Stable matters: a player must not oscillate between versions on every launch.
  • Rollout is the state machine over cohorts: 1 percent, 10, 50, 100, with a bake time and a health gate between each step, and a halt that stops it dead.
  • FeatureFlag is evaluated server-side per install and changes behaviour with no download at all.
  • Deployment is the server-side sibling of Rollout: waves across the fleet, with drain state per instance.

API or System Interface

GET    /v1/manifest?platform=&channel=&installId=&have=   // -> { version, chunks[], activateAt }
GET    /v1/flags?installId=&version=                     // -> { flagName: value }  // every session

POST   /v1/releases                     // { artifacts[] } -> promotes, never rebuilds
POST   /v1/rollouts                     // { releaseId, plan: [1,10,50,100] }
POST   /v1/rollouts/{id}/advance        // gated on bake time + health
POST   /v1/rollouts/{id}/halt           // global, under 60 s, no approval needed
POST   /v1/flags/{name}                 // { value, cohort }

POST   /v1/deployments                  // server fleet: canary, then waves
GET    /v1/deployments/{id}/health      // crash-free sessions, error rate, p99
  • The manifest call sends the chunks the client already has (have), so the service answers with a delta rather than a full list. This is what turns a 4 GB install into an 80 MB download.
  • activateAt says when the new version takes effect on the client.
  • Flags are fetched every session, not cached for a day.
  • Halt needs no approval and advance does. Stopping is always safe, so put no friction on it; proceeding is the risky direction, so gate that one.
  • Creating a Release takes artifact ids, never a source ref. The API makes rebuilding impossible rather than merely discouraged, which is how the build-once rule survives contact with a deadline.
  • Cohort comes from hashing the install id, not from a server-side assignment table. 200M rows of assignment is a database nobody needs, and a hash is stable across launches and updates.
    • Not across reinstalls: a reinstall usually mints a new install id, which is the point of one, so a player can land in a different cohort. Acceptable, and worth saying rather than claiming otherwise.
    • Keep a small override table anyway, for QA, dogfood, support escalations and allowlists. Pure hashing gives you no way to force one install into a cohort, and you will need that on day one.
DONE WHEN: you have said "flipping the manifest is the release" and "halt needs no approval, advance does".
STEP 3 OF 5

High-level design

10 min, end to end, no dives yet

1) Commit to artifact: build once, then only promote

CI: build the artifact once, then only ever promote it A developer opens a PR and CI builds it hermetically. Only the fast TierOne tests block the merge; TierTwo and TierThree run after merge. CI publishes the artifact to ArtifactStore under its content hash, ReleaseBuilder pulls artifacts by hash to stamp a Release, and the Release feeds both ServerRollout and ClientDelivery. CI: build the artifact once, then only ever promote it 2,000 commits a day, 500 engineers. PR feedback in under 10 minutes or people stop reading it. // per-PR wall clock, 2,000 commits a dayTierOne 8 min blocks the mergeTierTwo 35 min blocks the releaseTierThree 6 hrs blocks nothing Developer opens a PR1 of 500 engineers CI hermetic, cachedsame inputs, same bytes TestTiers same code, three depths, three cadences TierOne unit + lint, under 10 minruns on every PR TierTwo integration + protocol contractruns after merge TierThree soak, perf, full playthroughnightly, does not block why build once a rebuilt binary is a DIFFERENT binary.what you tested is then not what you shipped. so staging and prod run the same hash, andpromotion is a pointer move, not a build. ArtifactStore content-addressed by hashimmutable, never rebuilt ReleaseBuilder assembles a Releasefrom artifacts Release server image + client bundle + manifestone version number across all three   ServerImage ClientBundle Manifest ServerRollout we control it, 10,000 instancesnext diagram ClientDelivery players choose when, 200M installs4 GB build, 80 MB delta patch 1. Developer opens a PR and CI builds it hermetically 2. CI blocks the merge on TierOne only, so feedback stays under 10 minutes 3. CI runs the slower tiers after merge, where they cannot stall a review 4. CI publishes the built artifact under its content hash artifacts pulled by hash,never rebuilt 5. ReleaseBuilder stamps a Release from artifacts that already exist 6. The Release carries the server image and the client bundle together the not-so-happy path: the nightly tier finds what the fast tiers cannot, and it files a bug instead of blocking a queue.
  • Split tests into tiers so that only the fast tier, under 10 minutes, blocks a merge. The tiers protect the 10-minute number rather than being thorough for its own sake.
    • Tier one, unit and lint, under 10 minutes, blocks the merge. It is the only thing that blocks a human.
    • Tier two, integration and protocol contract tests, runs after merge where it cannot stall a review.
    • Tier three, soak and performance and a full playthrough, runs nightly and files bugs rather than blocking a queue.
    • With 2,000 commits a day, a 40-minute blocking suite means the merge queue never drains, so this is arithmetic rather than preference.
  • Build once and promote the same bytes, and make it structurally impossible to do otherwise.
    • A rebuilt binary is a different binary, so what you tested is not what you shipped.
    • Artifacts are content-addressed, so staging and production provably run the same hash.
    • Promotion is a pointer move, which also makes it instant and reversible on the server side.
  • The release bundles the server image and the client bundle together under one version, which is what makes the compatibility question answerable later instead of guessed at.

2) The server last mile: waves, gates, and draining a stateful fleet

Server rollout: waves, health gates, and draining a stateful fleet Deployer takes the immutable Release from CI and sends it to one canary instance. HealthGate watches the canary for a full bake, then the Deployer advances through waves of 5, 25 and 100 percent, each waiting for the gate to pass; Halt freezes the Deployer. Every instance in every wave is drained: no new matches, the live match finishes, then the instance is replaced. Server rollout: waves, health gates, and draining a stateful fleet 10,000 instances. game servers hold live matches, so you cannot just restart them. Release one immutable versionfrom the CI diagram Deployer advances waves,watches gates Canary 1 instance,20 min bake Wave1 5 percent500 instances Wave2 25 percent2,500 instances Wave3 100 percent10,000 instances Halt no approval needed,under 60 seconds HealthGate must pass beforethe next wave ReplacingOneInstance what a wave actually does, instance by instance Drainer no NEW matches to adraining instanceexisting matches finish GameServerInstance draining: no new matchesrouted here     LiveMatch finishes before theinstance is replaced InstanceReplaced restarts on the new versionrejoins the pool empty the same Release from CI,promoted, never rebuilt 1. Deployer sends the Release to a single canary instance first 2. HealthGate watches the canary for a full bake before anything moves 3. Deployer advances wave by wave, never all at once each wave waits for the gate to pass then then then each wave is bigger than the last, so a badbuild hits one instance's players, not 50M gate passes:the Deployer advances 6. Halt freezes the Deployer: no further waves move every instance in every wavegoes through this drain 4. Drainer stops routing new matches to an instance before it is replaced 5. The live match finishes, then the instance is replaced the not-so-happy paths: halting is not rolling back. halting stops the spread; rolling back is a second deploy with its own risk.one very long match can hold a draining instance open; the dive on deploying a stateful fleet handles the tail.
  • Canary first, one instance, with a real bake time. A canary you advance past in 30 seconds has told you nothing except that the process starts.
  • Then waves: 5 percent, 25 percent, 100 percent of 10,000 instances, with a health gate between each. The Deployer advances only when the health gate passes.
    • Which signals to trust, what to compare them against, and who halts when they fail is the dive on what stops a rollout.
  • Drain instances instead of restarting them, because a game server holds live matches. Draining is the part specific to this problem and worth spending time on.
    • A game server holds live matches, so you cannot restart it and you cannot move the match.
    • Draining means: stop routing NEW matches here, let existing ones finish, then replace the instance.
    • One very long match can hold a draining instance open; the dive on deploying a stateful fleet handles the tail.
  • Halting is not rolling back: halting stops the spread and is always safe. The distinction matters under pressure. Rolling back is a second deploy with its own risk, its own drain, and its own chance of being wrong.

3) The client last mile: cohorts and background downloads

Client delivery: a staged rollout, downloaded in the background GameClient asks ManifestService what it should be running, and the answer depends on the cohort its install id hashes into. The client downloads only the chunks it lacks from the CDN, in the background and resumable. ManifestService moves each cohort to the new version as the rollout advances through 1, 10, 50 and 100 percent. ArtifactStore publishes the Release's chunks to the CDN. Client delivery: a staged rollout, downloaded in the background 200M installs. the manifest decides what each install runs; the CDN serves the bytes. Cohorts 1, 10, 50, then 100 percentby hash of install id GameClient checks on launchand on a timer ManifestService answers: what should THIS install have?platform, channel, cohort, current version CDN serves the Release's chunksclose to players 1 GameClient asks ManifestService what it should be running 2 ManifestService answers from the cohort this install falls in 4 ManifestService moves a cohort to the new version when the rollout advances 3 GameClient downloads only the chunks it does not already have, in the background and resumable ArtifactStore the Release's chunks publisheschunks the step 2 answer, concretelyGET /v1/manifest?platform=ios&channel=live &installId=a91f&have=9c1e,07ab,...{ "version": "v43", "chunks": ["5d20...", ...], // only the chunks it lacks "activateAt": "..."} each cohort downloads when it advances, so the last step, 50 to 100 percent,still sends 2 PB as fast as players launch. the bandwidth dive handles it. the not-so-happy paths: - resumable and background is not a nicety: a player on mobile data who loses the connection at 79 MB must not start again.- never block launch on a download unless the protocol genuinely broke. a forced full-stop update is the worst thing you can do to a player.
  • The client asks the manifest service what to run, and the answer depends on its cohort.
    • The rollout advances cohort by cohort, 1, 10, 50, then 100 percent, and the manifest service moves each cohort to the new version as it advances.
    • The client downloads only the chunks it does not already have, from the CDN.
  • Downloads run in the background and resume where they stopped. That is not a nicety. A player on mobile data who drops at 79 of 80 MB must not start again, and the download must never hold the game hostage while they are trying to play.
  • Never block launch on a download unless the protocol genuinely broke, and be able to state exactly when that is true, because "we forced an update" is a retention event. The one honest case is a client older than N-2, which the server no longer speaks.
  • The client-side details that decide whether players actually like this:
    • Wi-Fi only by default with an explicit opt-in for cellular, because a silent 80 MB on a metered plan is a complaint.
    • A storage-full path that fails gracefully and says so. Holding two versions during an update is exactly when disks fill, and this is the most common real update failure.
    • Apply time, not just download time. On console, unpacking and installing a patch is often longer than fetching it, and it is what the player actually waits through.
  • Each cohort downloads the patch when it advances, so the last step, 50 to 100 percent, still sends 2 PB as fast as players launch; the bandwidth dive handles it.

4) Compatibility and the levers you actually have

Old clients keep working, and a bad release can be halted Three client versions are live at once and all talk to the same ServerAPI, which supports N-2 for 30 days. The server deploys first and backward compatible, and only then does RolloutController start the client rollout. Halting the rollout stops ManifestService from moving any new cohort to the patch, and repointing sends installs back to the previous Release. Old clients keep working, and a bad release can be halted several client versions are always live, so the server has to speak to all of them. GameClient installs, by version all live at the same time, for weeks v41 (12 percent) v42 (63 percent) v43 (25 percent) bar length is share of installs, drawn to scale ServerAPI speaks to all three versionsN-2 supported for 30 days ServerDeploy the new server version,accepts old and new RolloutController advance, hold, halt,or repoint the client rollout ManifestService the version each installis told to run every live version talksto the same server fleet updates 1 ServerDeploy finishes first, backward compatible; only then does the client rollout start 2 RolloutController halts, or repoints the manifest to the previous Release 3 ManifestService tells each install which version to run: the one it has, or the previous Release older than N-2 is the one case where aforced update is correct: the serverno longer speaks it. repointing does not help an install v43 already changed, for example by migrating its save data.the dive on un-shipping a client handles those players. the not-so-happy path: the server keeps accepting the old protocol while any supported client still speaks it; when it may stop is the version-skew dive.
  • Deploy the server first and backward compatible. Always. The reverse order breaks every client that has not updated, which is most of them on day one.
  • The server keeps accepting the old protocol while any supported client still speaks it. When the old path may be deleted, and how skew is tested, is the version-skew dive.
  • Skew has a user-visible cost the design has to own: the matchmaking pool splits. Three live client versions split the matchmaking pool, which means longer queues and worse matches for whoever is on the smallest version. The matchmaker is version-aware, not just the drainer, and that is the concrete price of the N-2 policy.
  • A bad release is stopped by halting its rollout.
    • On the server, halt freezes the waves, and the previous Release is a pointer move away.
    • On the client, halt means no new cohort receives the patch, and the manifest is repointed to the previous Release.
  • Repointing does not help an install the bad patch already changed, for example by migrating its save data; the dive on un-shipping a client handles those players.
DONE WHEN: all four requirements have a path, and you have said out loud which of the two last miles each risk belongs to.
STEP 4 OF 5

Potential deep dives

~20 min, interviewer steers
TRIGGER: "you shipped a bug to players" / "just roll it back"

1) You cannot un-ship a client

HALT STOPS THE SPREAD · THE FLAG STOPS THE DAMAGE · ONLY FORWARD FIXES IT
Bad Solution: roll back the client release
  • Repoint the release to the previous version, the way you would on a server.
    • For the launcher and asset layer this actually works, and claiming otherwise is the trap. The manifest is the release, chunks are content-addressed, and the old chunks are still on disk from the pre-download, so repointing to v42 is a cheap revert. Steam does exactly this by repointing a branch.
    • The thing that makes it a one-way door is local data, not bytes. If v43 migrated the save file or the local profile, v42 cannot read it, and a revert corrupts the player's progress rather than restoring it. That is the real reason a client revert is dangerous, and it is what an interviewer is waiting to hear.
    • For a STORE binary the mechanical objection also holds: the previous version is no longer served and replacing it needs review.
Good Solution: stage the rollout so only a few are affected
  • Start at 1 percent, watch, then advance. Correct and necessary, and it bounds the damage.
    • But 1 percent of 50M is still 500,000 players who now have the bug and cannot be given anything else.
    • Staging limits how many are hurt. It does nothing for the ones already hurt, and they are the ones filing reviews.
Great Solution: make the bad path switchable off from the server
  • Every risky change ships behind a flag, evaluated server-side per install on every session. Turning it off takes seconds and needs no download, no store, and no player action.
  • Turning the flag off is what "rollback" means for a client: you refuse to enable the code rather than remove it. Using the word precisely is most of this dive.
  • A manifest revert still works for the layer you deliver yourself (the launcher and asset layer); what you genuinely cannot un-ship is a store binary. The blunt claim that nothing can be reverted is wrong. The manifest revert is blocked by forward-only local data migrations rather than by the download. So: design migrations to be backward compatible for one version, and the manifest revert stays a real option.
  • Then flip the flag, halt the rollout, keep serving the previous manifest, and ship a hotfix forward, in that order. The flag stops the damage, the halt means nobody else receives the patch, the previous manifest goes to anyone who has not updated, and the hotfix is for the ones who have.
  • On mobile the store review clock makes the flag mandatory, because a binary fix is hours to days away and a flag is thirty seconds away. So the design pushes risky behaviour into data and configuration precisely so the fast lever exists.
  • Both stores do give you a halt, and pretending otherwise loses credibility. Play has staged rollout with a halt, Apple has phased release with a pause. What they do NOT give you is control of download timing, so the pre-download and manifest-flip trick applies to the launcher and asset layer, not to the store binary. On mobile the store serves the bytes and pays the egress, which also means the 4 PB bill below is the self-delivered layer's.
  • Fetch flags every session, not daily, and push revocations to sessions already running. A kill switch that takes a day to propagate is not a kill switch.
    • Per-session is still not "seconds" and I should not pretend it is. A player already in a 40-minute match keeps the bad path for up to 40 minutes.
    • So flag revocations are pushed down the live game socket, and anything genuinely dangerous is gated server-side per action rather than trusted to a client-cached value.
    • Seconds for new sessions, session-length for in-flight ones, unless you push.
    GET /v1/flags?installId=a91f&version=v43
    {
      "newMatchmaker": false,   // killed
      "seasonPassV2":  true,
      "chatFilterV3":  false
    }
  • Say the cost honestly: flags accumulate, every one is a branch, and two flags interact in ways nobody tested. So they expire: a flag that has been at 100 percent for two releases gets deleted, and that cleanup is scheduled work rather than good intentions.
  • And the thing a flag cannot save you from: a client that crashes on launch, or corrupts local data, before it ever reaches the flag service. That is what the canary cohort and a staged 1 percent are really insuring against, so both mechanisms are needed and they cover different failures.
TRIGGER: "the client and server both changed" / "what deploys first"

2) Version skew, and the order of operations

SERVER FIRST, BACKWARD COMPATIBLE · EXPAND THEN CONTRACT · N-2 FOR 30 DAYS
Bad Solution: ship them together
  • The client and server change in the same release, so deploy both at once.
    • They are not simultaneous in reality. The server flips in minutes; the clients take weeks.
    • For that whole window, unmigrated clients are talking to a server that no longer speaks their protocol, which is an outage for the majority of your players.
Good Solution: version the protocol and deploy the server first
  • The server handles v1 and v2, clients migrate, done. This is the right instinct and the right order.
    • What it leaves out is when the old path is allowed to be deleted, so in practice it is never deleted and the server accumulates every protocol it has ever spoken.
    • It also leaves skew untested: nobody runs the new server against an old client until a player does.
Great Solution: expand, migrate, contract, with the window written down
  • Expand: the server accepts both shapes and writes both. Nothing is removed and the change is invisible to old clients.
  • Migrate: clients update over the rollout. You watch the version distribution and you know exactly what fraction is still on the old shape, because the manifest service already knows what everyone is running.
  • Contract: the server deletes the old path only when the old version drops out of the support window, and deleting it is a scheduled task with a ticket, not a hope.
  • Write the support window down as a policy: N-2 (the current client and the two versions before it) for 30 days. Without a number, "support old clients" means forever, and the server becomes a museum.
  • Test the skew in CI, which is the part almost nobody does. Contract tests run the new server against the last N client protocol versions on every merge, so a breaking change fails a build rather than a player.
  • Name the tax rather than hiding it: every protocol change is written three times, and the middle state is live for a month. That is the actual price of not controlling your clients, and pretending otherwise is how skew bugs happen.
  • The same recipe governs the DATABASE, and that is the half people forget.
    • Expand the schema, dual-write, backfill at fleet scale, dual-read, then contract. Same shape, much slower, because the backfill is bounded by data size rather than by client adoption.
    • It also breaks the promotion story above: "promotion is a pointer move, so the server side is reversible" stops being true the moment a migration has run. A schema change is the point where roll-forward becomes the only direction on the server too.
    • So migrations ship separately from the code that needs them, at least one release ahead, and never in the same deploy.
TRIGGER: "50 million players need the patch" / "what does that cost"

3) The rollout is a bandwidth problem before it is a safety problem

4 PB · DOWNLOAD EARLY AND JITTERED · ACTIVATE ON A MANIFEST FLIP
Bad Solution: publish it and let everyone update
  • Push the notification, players update.
    • 50M times 80 MB is 4 PB. Concentrated in the hour after a patch note, that is 1.1 TB a second.
    • That is 8.9 Tbps. Buyable with committed capacity and notice, not on demand for one hour a fortnight and not at the marginal rate. In practice you get shaped and players see a patch that takes hours.
    • The bill is about 80,000 dollars either way, so this is not a small line item to be casual about.
Good Solution: percentage cohorts
  • Release to 1 percent, then 10, then 50, then 100, which spreads the load and limits blast radius.
    • It genuinely fixes the peak, and it is the same mechanism safety wants, which is convenient.
    • But it means players get the content at visibly different times, and for anything competitive or social that is a real fairness problem: a player in the 100 percent cohort is two days behind their friends.
Great Solution: decouple the download from the activation
  • Publish the bytes 48 hours before the release moment and let clients fetch them in the background, jittered across the window. 23 GB a second, an ordinary commitment.
  • Then flip the manifest, the server's answer to which version each install runs, and the clients that hold the bytes activate. Choose deliberately between two flips rather than pretending you get both.
    • A GLOBAL flip means everyone with the bytes activates at the same instant, which is what a content drop needs and what fixes the fairness problem cohorts create.
    • A STAGED flip gives each cohort its own activation moment, which is what a risky release needs, and it costs fairness.
    • So: stage the activation for anything that could break, flip globally for the content drop that has already baked behind a flag. Claiming simultaneous activation AND staged safety at once is the contradiction an interviewer will catch.
  • Pre-download solves bandwidth and staged activation solves safety, so the two mechanisms do different jobs. Conflating them is what makes people think they have to choose.
  • The chunk store is the delta mechanism, and precomputed binary diffs are an optimisation for the common hop (from the previous version straight to the new one). Deltas and chunking overlap more than people say.
    • The chunk store is primary: "give me the chunks I do not have" IS a delta against whatever state the client is in, from any version, with no N-way matrix.
    • Precomputed binary diffs are an optimisation for the common hop, not the mechanism. They are O(N) per release under N-2, so tens of objects, and the real cost is diffing CPU on a 4 GB build rather than storage.
  • Downloading a shared file once, ever, needs content-DEFINED chunking. With fixed-size chunks a one-byte insertion shifts every boundary after it and nothing dedupes, so boundaries come from a rolling hash, and pack files have to be chunk-aligned and per-chunk compressed or a small change reshuffles the whole archive.
  • Content-addressed chunks mean unchanged files are never re-sent, and two releases sharing a file share a cache entry globally.
  • Handle the client that did not pre-download, because a phone that was off for a week exists. It downloads on demand at activation, and that tail is small enough to absorb precisely because the bulk went early.
  • Confirm activation at flip time, because a timestamp alone activates on the client's clock even after a halt. This is the case the halt has to survive.
    • A client holding a cached manifest with a future activateAt will activate on its own clock even if the release was halted an hour ago, and a client with a wrong clock activates early.
    • So the client re-reads the manifest at flip time and the SERVER refuses the new protocol before T-0. The timestamp is a hint for when to check, never the authority.
    • Edge cache TTL on the manifest is then the real bound on "haltable in 60 seconds", so it is set in seconds, not minutes.
  • Say the operational cost: you are now holding two versions on disk on the client for up to 48 hours, and disk on a console or a phone is not free. That is a real constraint worth naming.
  • Encrypt the preload and ship the key at unlock, or dataminers read the content two days early. That is why pre-download in practice is encrypted rather than merely staged.
TRIGGER: "there are live matches on that server" / "how do you deploy without dropping people"

4) Deploying a stateful fleet without dropping sessions

DRAIN, DO NOT RESTART · A MATCH IS PINNED TO A VERSION · DEADLINE THE TAIL
Bad Solution: rolling restart
  • Replace instances one at a time and let the load balancer sort it out.
    • A stateless web request retries. A 40-minute match does not: the state is in memory on that instance and it is gone.
    • Every deploy becomes a visible outage for whoever was mid-match, and with 10,000 instances that is a lot of people per release.
Good Solution: drain before replacing
  • Mark the instance draining, stop sending new matches, wait for the existing ones to end. This is the right mechanism.
    • The gap is the tail: one very long match, or one idle lobby nobody leaves, blocks the instance forever.
    • Without a deadline, a deploy can simply never finish, and the deploy that never finishes is the one that blocks the security patch behind it.
Great Solution: drain with a deadline, and pin the match to a version
  • A match runs one server version end to end. Mixing versions mid-match means two players are running different rules, which is worse than a disconnect.
  • Drain each instance (no new matches, live ones finish), so drain time is bounded by match length and you budget for it. It is not a surprise. If matches are 40 minutes, a full fleet rollout is measured in hours and the plan says so.
    • Canary 20 min, then 15 min a wave, plus drain time on every wave: 20 + 40 + 3 × (15 + 40) = 225 min, about 4 hours with 40 min matches.
  • The tail gets a hard deadline plus reconnect. Past the deadline the instance goes anyway, clients reconnect into a new match or a recovery flow, and you have chosen a small known harm over an unbounded stall.
  • A ranked match killed at the drain deadline is voided and its entry refunded. Say what happens to the RESULT and not just the connection, because "they reconnect" is not an answer in competitive play.
  • Capacity has to lead the drain: spare new-version instances must be up before the old ones stop accepting. Otherwise you drain straight into a shortage at peak. That is the mistake that turns a safe deploy into a queue of players who cannot get a match.
  • Pinning a match to one version is a default, not physics. Snapshot-and-migrate between processes exists; it is rejected on cost and on the risk of moving live state, and saying so is better than claiming it cannot be done.
  • Bounded matches are the easy case. Persistent worlds and social hubs have no natural boundary, so they need scheduled shard downtime, player transfer to another shard, or a hot-reloadable script layer for anything that must change between windows.
  • Deploy when your players are asleep, per region, which is free if the fleet is regional and is the cheapest risk reduction available.
  • Emergency path is different and should be stated separately: for a security fix you accept dropping matches, because the tradeoff has changed. Having decided that in advance is what makes it executable at 3am.
TRIGGER: "who decides to halt" / "how do you know it is bad"

5) What stops a rollout, and how fast

CRASH-FREE SESSIONS · BAKE PER STAGE · AUTOMATIC HALT, HUMAN RESUME
Bad Solution: watch the dashboards
  • Someone keeps an eye on it and calls it if something looks wrong.
    • At 1 percent of 50M, a crash affecting one player in a thousand is 500 people before anyone finishes a coffee.
    • Nobody watches at 2am, and 2am in one region is prime time in another.
Good Solution: alert on error rate
  • Page someone when errors cross a threshold. Better, and it at least fires at 2am.
    • Server error rate is the wrong signal for a client bug: a client that crashes on launch generates fewer server errors, not more, because it never connects.
    • Paging a human still costs the minutes it takes them to wake up and decide, and the rollout keeps advancing while they do.
Great Solution: gate on client-side health, and make halting automatic
  • Crash-free session rate is the primary gate, measured per version and per platform, because it is the one metric that moves for exactly the failure you are afraid of.
  • Compare against the cohort still on the old version, not against an absolute number. Crash rates vary by day, region and device, so the previous version running concurrently is the only honest baseline. Staged rollout gives you that control group for free.
    • Breach when crash-free sessions fall more than 0.2 pp below the old-version cohort.
    • Breach when error rate rises above 1.2x the old-version cohort.
    • Breach when p99 latency rises above 1.3x the old-version cohort.
  • Each stage bakes (waits before advancing) for as long as the failure it guards against takes to show. A crash shows in minutes, so a 20-minute canary catches those, but a leak or a progression bug takes hours and needs an overnight canary before the first real wave. Early stages bake longest, and advancing fast at 1 percent defeats the point of having a 1 percent.
  • Halt automatically, resume manually. Stopping is safe and should need no human; continuing is the risky direction and should need a person who has looked.
  • Watch the metrics that only exist on the client: crash-free sessions, launch success, download success and time-to-first-input. A rollout can be perfectly healthy server-side while the client is unusable, and only client telemetry sees it.
  • Give the manifest and flag services, which every launch and session calls, their own failure policy, because they are the real single points of failure. The manifest service fails OPEN to last-known-good, since failing closed stops 200M installs from launching. The flag service falls back to the last cached value with a short TTL and a safe default per flag, since a kill switch that fails open is not a kill switch. Both sit behind the edge cache, and neither is a hard dependency of starting the game.
  • Cover the case where telemetry itself is what broke: a sudden DROP in reported sessions is an alarm, not silence. A client that cannot start cannot tell you it cannot start, and treating zero as good is how a bad rollout goes to 100 percent.
  • Be honest about the detection window. A launch crash is reported on the NEXT launch, so with any aggregation the automatic halt is minutes, not 60 seconds. The 60 seconds is how fast a halt PROPAGATES once decided, and conflating the two oversells the system.
  • State the statistics rule or the gate halts on noise. Early cohorts are not a random sample, so bucket by device, OS and region, and require a minimum sample and a significance threshold before a breach counts.

Four more dives, briefly

  • Sign manifests and artifacts and verify them on the client, because whoever forges a manifest owns 200M installs. Signing is the thing a CI/CD question is really about, and this design has so far left it open.
    • Content addressing gives integrity only relative to a manifest you already trust. Whoever forges a manifest response owns 200M installs, which makes this the highest-value attack surface in the company.
    • So manifests and artifacts are signed, the client pins a root key and verifies before applying, and the signing key lives in an HSM rather than a CI environment variable.
    • Build provenance is the other half: record what source, what toolchain and what dependencies produced each hash, so a compromised dependency is traceable rather than theoretical.
    • Secrets never enter an artifact. They are fetched at run time by identity, which is also what lets the same immutable image run in staging and production.
  • Make the pipeline itself trustworthy with hermetic builds, merge queues and same-day flaky-test quarantine. That is what makes everything else possible.
    • Hermetic, cached builds so the same input gives the same bytes, which is also what makes content addressing meaningful.
    • Merge queues that test the combination rather than each PR alone, because two PRs that pass separately can break together.
    • Flaky tests quarantined the same day, with an owner and a deadline. A gate people have learned to ignore is worse than no gate, because it costs time and buys nothing.
    • Track pipeline p50 and p95 as a product metric owned by someone. It regresses silently, one minute at a time, until nobody can ship on a Friday.
  • Artifact retention is coupled to the support window, so it is not a housekeeping detail. Every chunk any N-2 client might still need has to stay alive, which means garbage collection is driven by the 30-day policy rather than by disk pressure, and deleting a chunk early is how you brick the clients that had not updated.
  • Price the CI side too. 2,000 commits a day against a 4 GB build is the other budget line, and it is what makes hermetic caching a cost decision as much as a correctness one.
  • Ship data, tuning and content as configuration, which skips store review, and give it the same staged rollout as code. It is the release valve for everything the store slows down.
    • Split what ships: executable code needs review; data, tuning, and content packs do not.
    • So balance changes, event schedules and drop rates go out as data in minutes, while the binary keeps its slow, safe cadence.
    • The catch to name: data can break a client just as thoroughly as code, so config gets the same staged rollout and the same health gates. Treating it as safe because it is "just data" is how you take down a client with a JSON file.
  • Console and mobile stores have review and PC and web do not, so the N-2 window is sized against the slowest platform. Multi-platform means the pipeline is not one pipeline.
    • Console and mobile stores have review; PC and web do not. So the same release lands on different platforms at different times, and the server has to be compatible with all of them at once.
    • That widens the skew window rather than changing its nature, and the N-2 policy has to be sized against the slowest platform, not the fastest.
    • Certification failure is a scheduling risk with no engineering fix, so the release plan carries a buffer and the feature is behind a flag either way.
DONE WHEN: you have opened two or three dives properly, and "I cannot un-ship a client" has been used as a REASON at least twice rather than restated as a fact.
FLASHCARDS · THE FIVE HARDEST PROBESshow all (interview mode)
You shipped a client bug to the 10 percent cohort. What do you actually do?
  • Not a rollback. A player who installed v43 has v43 on disk, and repointing a pointer does nothing for them.
  • Flip the flag first. Risky code ships behind a server-side flag, so turning the path off takes seconds and needs no download, no store and no player action.
  • Halt the rollout so nobody else receives the patch, and keep serving the previous manifest to anyone who has not updated.
  • Then ship a hotfix forward, because forward is the only direction that exists for the players already on it.
  • On mobile the store review clock is why the flag is not optional: a binary fix is hours to days, a flag is thirty seconds.
  • What a flag cannot save: a client that crashes on launch before it ever reaches the flag service. That is what the 1 percent cohort insures against, which is why you need both.
The client and the server both change the protocol. What order, and why?
  • Server first, and backward compatible. The reverse order breaks every client that has not updated, which on day one is most of them.
  • Expand, migrate, contract: the server accepts both shapes, clients migrate over weeks, and only then is the old path deleted.
  • The window is a written policy, N-2 for 30 days. Without a number, "support old clients" means forever and the server becomes a museum.
  • Contract tests in CI run the new server against the last N client protocol versions, so skew breaks a build instead of a player.
  • Name the tax: every protocol change is written three times and the middle state is live for a month. That is the price of not controlling your clients.
50 million players need an 80 MB patch. Do the math and tell me what breaks.
  • 50M times 80 MB is 4 PB. In one hour that is 1.1 TB a second, which nobody sells on demand for one hour a fortnight.
  • Spread over 48 hours it is 23 GB a second, an ordinary CDN commitment. So staged delivery is a bandwidth requirement before it is a safety mechanism.
  • It also costs about 80,000 dollars per rollout of deltas at two cents a gigabyte, so delta patching is a budget line, not an optimisation.
  • The fix is to separate download from activation: publish 48 hours early, fetch in the background jittered, then flip the manifest so everyone activates at once.
  • That also solves the fairness problem staged cohorts create, where a player is two days behind their friends.
  • Content-addressed chunks mean unchanged files are never re-sent, and the client that was switched off all week downloads on demand as a small tail.
How does a live match survive a server deploy?
  • By draining rather than restarting: stop routing new matches to the instance, let the existing ones finish, then replace it.
  • A match runs one server version end to end, because mixing versions mid-match means two players are running different rules, which is worse than a disconnect.
  • So drain time is bounded by the longest match, not by process shutdown. If matches are 40 minutes, a full fleet rollout is hours and the plan says so.
  • The tail gets a hard deadline plus reconnect, or one idle lobby stalls the deploy forever, and that stalled deploy is blocking the next security patch.
  • Capacity leads the drain: you need the new instances up before the old ones stop accepting, or you drain into a shortage at peak.
  • For a security fix the tradeoff flips and you accept dropping matches. Deciding that in advance is what makes it executable at 3am.
What automatically stops a rollout, and how fast?
  • Crash-free session rate per version and per platform is the primary gate, because it moves for exactly the failure you fear.
  • Server error rate is the wrong signal for a client bug: a client that crashes on launch produces fewer server errors, not more.
  • Compare against the cohort still on the old version rather than an absolute threshold, since crash rates vary by day, region and device. Staged rollout gives you that control group for free.
  • Bake time is sized to the signal: crashes show in minutes, leaks and progression bugs take hours, so early stages bake longer than late ones.
  • Halt is automatic and needs no approval; resume is manual and needs a person who has looked. Stopping is always the safe direction.
  • A sudden drop in reported sessions is an alarm, not silence: a client that cannot start cannot tell you it cannot start.
STEP 5 OF 5

Final design + what is expected at each level

wrap

Final Design

Commit to player: one pipeline, two very different last miles CI builds each commit once and publishes it to ArtifactStore by content hash; ReleaseBuilder reads those hashes and promotes the same bytes into a Release. The server image goes to the Deployer, which rolls it across the ServerFleet canary first while HealthGate can halt it and the Drainer lets live matches finish. The client bundle goes to PatchBuilder and the CDN, and the manifest to ManifestService, which GameClients ask before downloading chunks in the background. RolloutController flips the cohort, FlagService and KillSwitch turn a bad path off with no download, and a binary update waits on AppStoreReview. Commit to player: one pipeline, two very different last miles 2,000 commits a day. 10,000 server instances you control, 200M client installs you do not. BUILDbuild it once, hash it, and promote the same bytes forever Developer CI hermetic, cachedTierOne blocksmerge, under 10 min ArtifactStore content-addressed,immutablebuilt once, never rebuilt ReleaseBuilder picks the hashes,writes the manifest Release server image + clientbundle + manifestone version, promotednot rebuilt 1. Developer merges and CIbuilds the artifact once 2. CI publishes it under itscontent hash, and nothingrebuilds it again ReleaseBuilder reads thehashes, copies nothing 3. ReleaseBuilder promotes the samebytes into a Release what a Release pins, by hash and nothing else: { "version": "v43", "builtFrom": "commit 8f2a1c", "serverImage": "sha256:9c1e4b...", "clientBundle": "sha256:04bb7f...", "supersedes": "v42" } SERVER LAST MILEyou control the timing Deployer canary, then 5, 25,100 percent ServerFleet 10,000 instances HealthGate crash-free sessions,error rate, p99auto-halt in under 60 s Drainer no new matches,existing ones finish 4. Deployer rolls the server outcanary first, wave by wave 6. HealthGate haltsthe rollout the momenta gate breaks reports what thegates measure 5. Drainer lets livematches finish on theversion they started on what halts a wave, vs the cohort still on the old version: crash-free sessions more than 0.2 pp below iterror rate above 1.2x its ratep99 above 1.3x its p99 CLIENT LAST MILEthe player controls the timing ManifestService what should THIS install run?cohort, platform, channel PatchBuilder binary delta persupported version4 GB full becomes 80 MB GameClient background, resumable, jitteredv41, v42 and v43 all live CDN content-addressed chunks4 PB per full rollout 8. GameClient asks ManifestServicewhat it should be running the answer names the version and its chunks 7. PatchBuilder precomputes a deltafrom every version still supported 9. GameClient downloads chunks in thebackground, days before activation what ManifestService answers one install: { "version": "v43", "chunks": ["5d20...", ...], "activateAt": "2026-08-19T17:00Z" } // chunks: the v42-to-v43 delta, 80 MB CONTROLthe instant levers RolloutController halt, hold, advance FlagService server-side, per installthe only leverwith no download AppStoreReview hours to days, not ours KillSwitch off in seconds one flag set to off, no rollout needed how fast each lever really is: flag or kill switch: seconds, no downloadhalt or hold the cohort: under 60 sroll forward a client: hours to days the manifest for this version the client bundlefor this version the server imagefor this version 10. RolloutController flips the cohort, andactivation is instant for everyone in it 11. FlagService turns a bad path offwithout shipping anything at all a binary update waits here, a data-only patch does not Deploy the server first and backward compatible. Three client versions are live for weeks, so every protocol change is carried N-2.You cannot un-ship a client. Halting stops the spread, the flag stops the damage, and only a roll-forward actually fixes it.
  • CI builds hermetically and only tier one blocks a merge, so 2,000 commits a day still get feedback in under 10 minutes.
  • Artifacts are content-addressed and immutable, a Release promotes the same bytes, and creating one takes artifact ids rather than a source ref so rebuilding is impossible rather than discouraged.
  • The server rolls out canary then 5, 25 and 100 percent across 10,000 instances, with a health gate that halts automatically, and a drainer that lets live matches finish on the version they started on.
  • The patch builder precomputes a delta from every supported version, and the CDN serves content-addressed chunks so an unchanged file is never sent twice.
  • Clients fetch in the background 48 hours early and jittered, then the manifest flips and everyone in the cohort activates at once: 23 GB a second instead of 1.1 TB a second.
  • The flag service is evaluated server-side every session and is the only lever that changes behaviour with no download at all.
  • The invariant worth closing on: the server is a rollout problem and the client is a delivery problem, because I control one timeline and not the other. Server first and backward compatible, download early and activate late, and flags instead of rollback.

What is Expected at Each Level

  • Mid:
    • Builds a working pipeline: CI, artifact storage, environments, automated deploy with a canary.
    • Knows to stage a rollout and to have a rollback plan.
    • Usually treats the client like a server and assumes a rollback is available.
    • Usually misses that several client versions are live at once, so the compatibility question never comes up.
  • Senior:
    • Separates the two last miles and gets the deploy order right, server first and backward compatible.
    • Designs delta patching and staged cohorts, and handles draining a stateful fleet rather than restarting it.
    • Uses feature flags as the client-side safety mechanism and gates rollout advancement on health.
    • Sizes the test tiers against the commit rate instead of testing everything everywhere.
    • On the deep dives:
      • You cannot un-ship a client: puts risky client code behind feature flags and stages the client rollout at 1 percent first.
      • Version skew: versions the protocol so the server handles old and new, and deploys the server first.
      • Bandwidth: spreads delivery over 1, 10, 50 and 100 percent cohorts to cut the peak, and names the cost, players getting the content days apart.
      • Stateful fleet: drains each instance before replacing it (no new matches, live ones finish), and sees that one long match can stall the deploy.
      • What stops a rollout: gates each stage's advance on a health threshold rather than on someone watching dashboards, so it holds at 2am too.
  • Staff:
    • Does the 4 PB arithmetic unprompted and concludes that staging is a bandwidth requirement, not only a safety one.
    • Separates download from activation, and can say why that specific move solves both the peak and the fairness problem at once.
    • Says plainly that a client rollback does not exist, and reframes the levers as halt, flag and roll forward.
    • Treats store review as a design force, pushing risky behaviour into config so the fast lever exists at all.
    • Writes the skew window down as a policy with a number, and tests skew in CI rather than discovering it in production.
    • Picks client-side health as the gate and knows that a drop in telemetry is an alarm rather than silence.
    • Names what the design does not do: flags accumulate and interact, two versions sit on client disk during pre-download, and a launch-crash bug is caught by the cohort rather than by any lever.
DONE WHEN: you can point at the final diagram and say, in one breath, build once and promote, server first and backward compatible, download early and activate late, flags instead of rollback, and halt automatically.