← all topics Client CI/CD board notes · say it in this order 1 Problem2 Entities + API 3 High-level design4 Deep dives 5 Final + levelsreset

Design an end-to-end CI/CD pipeline for client updates (notes)

2,000 commits a day · 10,000 server instances · 200M client installs · Hello Interview flow, board rules, one idea per line
SAY THIS SENTENCE FIRST

"There are two last miles here and only one of them is mine."

Server: I control when it changes -> rollout, canaries + health gatesClient: I do not -> several versions live at once, never un-shippedSo:server deploys first, backward compatiblepatch downloaded days before activationonly instant lever on shipped code: server-side flag
STEP 1 OF 5

Understanding the problem

5 min

Functional Requirements

  1. Merged commit -> production automatically, through gates.
  2. Client patch -> staged rollout, background, resumable.
  3. Bad release stoppable fast, server and client.
  4. Old clients keep working mid-rollout.

Below the line (out of scope):

engine + asset build system (only need: deterministic artifact)cheat detection, telemetry schemas, localisation pipelinedeveloper environment tooling, local iterationStore review IN scope as constraint: hours to days -> risky behaviour ships as config, not binary

Non-Functional Requirements

Client bandwidth:50M daily × 80 MB patch = 4 PB egress per releasein 1 h: 1.1 TB/s - not buyableover 48 h: 23 GB/s - ordinary CDNstaged delivery: bandwidth requirement before safety mechanismCost:~$0.02/GB × 4 PB deltas = ~$80,000full 4 GB × 50M = 200 PB = ~$4Mdelta patching: budget lineall 200M installs: × 4Version skew:3+ client versions live at once, for weekssupport N-2 >= 30 days, every protocol change carried 3×Pipeline speed: PR feedback < 10 min, merge -> canary < 1 h500 engineers, 2,000 commits/dayClient patch: background, resumable, never blocks launch unless protocol brokeHalt: global <= ~60 s; bad code path off by flag, no downloadServer deploys: no dropped sessions; match pinned to one version -> drain time = match lengthArtifacts: immutable, content-addressed; tested bit-identical to shipped; promotion = pointer move
DONE WHEN: interviewer has heard "4 PB, so staging is a bandwidth requirement" and "I cannot un-ship a client".
STEP 2 OF 5

Entities + API

5 min

Defining the Core Entities

Artifact: {hash, kind: server image | client exe | pack file, bytes}immutable, name is the content hashRelease: {releaseId, version, serverImage, clientBundle, manifest}one version across all threeManifest: {version, platform, channel, cohort, chunks[] (content-addressed), activateAt}"flipping the manifest is the release"Channel: internal dogfood | alpha | beta | livestage before the 1% cohort, ~1 week on employeesCohort: {channel, bucket = hash(installId)}stable across launchesRollout: {releaseId, plan: [1, 10, 50, 100], step, bakeTime, healthGate, halted}FeatureFlag: {name, value, cohort}, server-side per install, no downloadthe client's rollbackDeployment: {releaseId, waves[], instance: drain state}

API or System Interface

GET    /v1/manifest?platform=&channel=&installId=&have=   // -> { version, chunks[], activateAt }
GET    /v1/flags?installId=&version=                     // -> { flagName: value }  // every session

POST   /v1/releases                     // { artifacts[] } -> promotes, never rebuilds
POST   /v1/rollouts                     // { releaseId, plan: [1,10,50,100] }
POST   /v1/rollouts/{id}/advance        // gated on bake time + health
POST   /v1/rollouts/{id}/halt           // global, under 60 s, no approval needed
POST   /v1/flags/{name}                 // { value, cohort } -> the instant lever

POST   /v1/deployments                  // server fleet: canary, then waves
GET    /v1/deployments/{id}/health      // crash-free sessions, error rate, p99
have=: server answers with a delta, 4 GB install -> 80 MB downloadactivateAt: download now, activate laterhint, not authority: client re-reads manifest at flip, server refuses new protocol before T-0why: cached manifest + halt, wrong clockmanifest edge cache TTL in seconds, bounds "haltable in 60 s"Flags: every session, never cached a dayin-flight match keeps bad path <= match length (~40 min)-> push revocations down the game socket; dangerous actions gated server-sideHalt: no approval. Advance: gated.POST /releases: artifact ids only, never a source ref, rebuild impossibleCohort: hash(installId), no 200M-row assignment tablereinstall -> new installId -> new cohort, acceptablesmall override table: QA, dogfood, support, allowlists
DONE WHEN: said "flipping the manifest is the release" and "halt needs no approval, advance does".
STEP 3 OF 5

High-level design

10 min, end to end, no dives yet

1) Commit to artifact: build once, then only promote

CI: build the artifact once, then only ever promote it A developer opens a PR and CI builds it hermetically. Only the fast TierOne tests block the merge; TierTwo and TierThree run after merge. CI publishes the artifact to ArtifactStore under its content hash, ReleaseBuilder pulls artifacts by hash to stamp a Release, and the Release feeds both ServerRollout and ClientDelivery. CI: build the artifact once, then only ever promote it 2,000 commits a day, 500 engineers. PR feedback in under 10 minutes or people stop reading it. // per-PR wall clock, 2,000 commits a dayTierOne 8 min blocks the mergeTierTwo 35 min blocks the releaseTierThree 6 hrs blocks nothing Developer opens a PR1 of 500 engineers CI hermetic, cachedsame inputs, same bytes TestTiers same code, three depths, three cadences TierOne unit + lint, under 10 minruns on every PR TierTwo integration + protocol contractruns after merge TierThree soak, perf, full playthroughnightly, does not block why build once a rebuilt binary is a DIFFERENT binary.what you tested is then not what you shipped. so staging and prod run the same hash, andpromotion is a pointer move, not a build. ArtifactStore content-addressed by hashimmutable, never rebuilt ReleaseBuilder assembles a Releasefrom artifacts Release server image + client bundle + manifestone version number across all three   ServerImage ClientBundle Manifest ServerRollout we control it, 10,000 instancesnext diagram ClientDelivery players choose when, 200M installs4 GB build, 80 MB delta patch 1. Developer opens a PR and CI builds it hermetically 2. CI blocks the merge on TierOne only, so feedback stays under 10 minutes 3. CI runs the slower tiers after merge, where they cannot stall a review 4. CI publishes the built artifact under its content hash artifacts pulled by hash,never rebuilt 5. ReleaseBuilder stamps a Release from artifacts that already exist 6. The Release carries the server image and the client bundle together the not-so-happy paths: a flaky test that blocks merges gets quarantined the same day, because the alternative is engineers learning to ignore red.the nightly tier finds what the fast tiers cannot, and it files a bug instead of blocking a queue.
Test tiers: protect the 10-minute numbertier 1: unit + lint, < 10 min, blocks mergetier 2: integration + protocol contract, post-mergetier 3: soak + perf + full playthrough, nightly, files bugswhy: 2,000 commits/day × 40 min suite, queue never drainsBuild once, promote the same bytes:content-addressed, staging and prod same hashpromotion: pointer move, instant, reversibleRelease: server image + client bundle, one versionFlaky tests: quarantined same day

2) The server last mile: waves, gates, and draining a stateful fleet

Server rollout: waves, health gates, and draining a stateful fleet Deployer takes the immutable Release from CI and sends it to one canary instance. HealthGate watches the canary for a full bake, then the Deployer advances through waves of 5, 25 and 100 percent, each waiting for the gate to pass; a broken gate halts the rollout. Every instance in every wave is drained: no new matches, the live match finishes on its version, then the instance is replaced. Server rollout: waves, health gates, and draining a stateful fleet 10,000 instances. game servers hold live matches, so you cannot just restart them. Release one immutable versionfrom the CI diagram Deployer advances waves,watches gates Canary 1 instance,20 min bake Wave1 5 percent500 instances Wave2 25 percent2,500 instances Wave3 100 percent10,000 instances Halt stops the rolloutin under 60 seconds HealthGate crash-free sessions, error rate, p99auto-halt on breach ReplacingOneInstance what a wave actually does, instance by instance Drainer no NEW matches to adraining instanceexisting matches finish GameServerInstance draining: no new matchesrouted here     LiveMatch a match must run ONE serverversion end to end InstanceReplaced restarts on the new versionrejoins the pool empty the same Release from CI,promoted, never rebuilt 1. Deployer sends the Release to a single canary instance first 2. HealthGate watches the canary for a full bake before anything moves 3. Deployer advances wave by wave, never all at once each wave waits for the gate to pass then then then each wave is bigger than the last, so a badbuild hits one instance of players, not 50M gate passes:the Deployer advances Halt freezes the Deployer:no further waves move 6. HealthGate halts the whole rollout the moment a gate breaks every instance in every wavegoes through this drain 4. Drainer stops routing new matches to an instance before it is replaced 5. LiveMatch finishes on the version it started on, then the instance goes // HealthGate, per wave, vs the concurrent holdoutcrash_free_sessions >= holdout - 0.2pperror_rate <= 1.2x holdoutp99_latency <= 1.3x holdoutbake 20 min canary, overnight if leak-suspect how long a full rollout takes canary 20 min, then 15 min a waveplus drain time on every instanceabout 4 hours with 40 min matches A match pinned to one version is why draining takesmatch-length, not seconds. Budget the longest match,and set a hard deadline with reconnect for the tail. the not-so-happy paths: halting is not rolling back. halting stops the spread; rolling back is a second deploy with its own risk.a drain that never finishes is a stuck deploy, so the deadline is mandatory and the tail reconnects.
Canary: 1 instance, real bake timeWaves: 5% -> 25% -> 100% of 10,000 instanceshealth gate each step: crash-free sessions, error rate, p99auto-haltDrain: stop routing new matches -> existing finish -> replace instancea match runs one server version end to enddrain time bounded by longest match, minutestail: hard deadline + reconnectHalt: stops spread, always safeRollback: second deploy, own drain, own risk

3) The client last mile: download now, activate later

Client delivery: separate the download from the activation The patch is published 48 hours early and clients fetch it in the background; at T-0 the manifest flips and everyone in the cohort activates at once. GameClient asks ManifestService what to run, gets an answer from its cohort, and downloads only the chunks it lacks from the CDN. PatchBuilder writes deltas to ArtifactStore, which publishes chunks to the CDN once. Client delivery: separate the download from the activation 50M players x 80 MB is 4 PB. delivered in one hour that is 1.1 TB a second, which is not a thing you can buy. download is spread over two days; activation is instant for the whole cohort. T-48h patch published,download starts quietly T-48h to T-0 clients fetch in the background, jittered T-0 manifest flips,the cohortactivates at once T+0 the cohort target the same patch delivered on activation day:4 PB in one hour, 1.1 TB/sec. staged delivery is aBANDWIDTH requirement, not only a safety one. every cohort gets its own T-0. the 1 percent cohort flips two days beforethe 100 percent cohort does. Cohorts 1 percent, 10 percent,50 percent, 100 percent GameClient checks on launchand on a timer ManifestService answers: what should THIS install have?platform, channel, cohort, current version CDN content-addressed chunksa chunk is cached once, globally 1 GameClient asks ManifestService what it should be running 2 ManifestService answers from the cohort this install falls in 5 ManifestService flips the active version when the cohort advances 3 GameClient downloads only the chunks it does not already have, in the background and resumable the step 2 answer, concretelyGET /v1/manifest?platform=ios&channel=live &installId=a91f&have=9c1e,07ab,...{ "version": "v43", "chunks": ["5d20...", ...], // the 1,284 chunks it lacks "deltaBytes": 83886080, "activateAt": "T-0"} PatchBuilder binary diff from eachsupported version4 GB full becomes an 80 MB delta ArtifactStore 4 PatchBuilder precomputes a delta from every supported version, so nobody re-downloads 4 GB chunks are publishedhere once, thencached at the edge the not-so-happy paths: - resumable and background is not a nicety: a player on mobile data who loses the connection at 79 MB must not start again.- a chunk is addressed by content, so two patches that share a file share a cache entry.- never block launch on a download unless the protocol genuinely broke. a forced full-stop update is the worst thing you can do to a player.
Separate delivery from activation:publish patch 48 h early, background fetch, jitteredflip manifest -> clients with bytes activate4 PB / 48 h = 23 GB/s - buyable4 PB / 1 h = 1.1 TB/s - impossibleDelta patching: 4 GB client -> 80 MB downloadpatch builder precomputes diff per version in the fieldContent-addressed chunks: shared file downloaded once, CDN caches by contentBackground + resumable: drop at 79 of 80 MB, no restartForced update: only client < N-2, server no longer speaks it"we forced an update" is a retention eventClient details:Wi-Fi only by default, cellular opt-instorage-full path, two versions held during preloadapply time (console unpack + install)encrypt preload, key at unlock (dataminers)

4) Compatibility and the levers you actually have

You cannot un-ship a client, so flags are the rollback Three client versions are live at once and all talk to the same ServerAPI. Each session the client asks FlagService what is enabled, so FlagService can switch a bad code path off in seconds. RolloutController halts the cohort through ManifestService, which keeps serving the previous version, and Engineering ships a HotfixRelease forward through app store review. You cannot un-ship a client, so flags are the rollback several client versions are always live. the only thing you can change instantly is what the server tells them. GameClient installs, by version all live at the same time, for weeks v41 (12 percent) v42 (63 percent) v43 (25 percent) bar length is share of installs, drawn to scale FlagService evaluated server-side, per installthe ONLY instant lever KillSwitch turns the new code path offtakes effect in seconds, no download no client update is involved:the next session sees the new value. RolloutController halt, hold, or advance a cohort ManifestService the version each installis told to run ServerAPI must speak to all threeN-2 supported for 30 days older than N-2 is the one casewhere a forced update is correct:the server no longer speaks it. AppStoreReview hours to daysthe reason risky logic ships asCONFIG, not as binary HotfixRelease roll FORWARD, the only real fix Engineering 1 GameClient asks FlagService what is enabled for it, on every session 2 FlagService turns the bad code path off without shipping anything every live version talks tothe same server fleet 3 RolloutController halts the cohort so nobody else receives the patch 4 ManifestService keeps serving the previous version to anyone who has not updated a binary hotfixstill waitsfor review 5 Engineering ships a HotfixRelease forward, because the players who already patched cannot go back EXPAND / CONTRACT: how a protocol change ships safely 1 server adds the new field,still accepts the old 2 clients migrateover weeks 3 server drops the old field,only after N-2 ages out deploy the server FIRST and backward compatible, always. the reverseorder breaks every client that has not updated yet. Rollback is a server concept. A player who already installed v43 HAS v43.The flag is what saves you; the hotfix is what fixes you. how fast each lever isflag flip secondscohort halt under 60 shotfix binary hours to daysclient re-download hours to days what FlagService returns in step 1GET /v1/flags?installId=a91f &version=v43{ "newMatchmaker": false, <- killed "seasonPassV2": true, "chatFilterV3": false} the not-so-happy paths: - the store review clock is why anything risky is a flag or a data file, never a binary change.- N-2 for 30 days is a real cost: every protocol change is carried three times over.
Order: server first, backward compatible, alwaysExpand then contract, over weeks:server adds new field, still accepts oldclients migrate over the N-2 windowold version ages out -> server drops old fieldcontract tests in CI: new server × last N client protocolsSkew cost: 3 live versions × one server version per match = 3 matchmaking poolslonger queues, worse matches on smallest version; matchmaker version-awareFlags: the client's rollbackrisky code behind a flag by default; store review off critical pathLevers, in order: flip flag (seconds, no download) -> halt rollout -> hotfix forward (only real fix)
DONE WHEN: all four requirements have a path; each risk assigned to one of the two last miles.
STEP 4 OF 5

Potential deep dives

~20 min, interviewer steers
TRIGGER: "you shipped a bug to players" / "just roll it back"

1) You cannot un-ship a client

HALT STOPS THE SPREAD · THE FLAG STOPS THE DAMAGE · ONLY FORWARD FIXES IT
Answer: flag off (seconds) -> halt rollout -> previous manifest -> hotfix forwardun-shippable: the store binary only
Bad Solution: roll back the client release
Launcher / asset layer: revert worksmanifest is the release, chunks content-addressed, old chunks on disk -> repoint to v42 (Steam: branch repoint)One-way door: local datav43 migrated save -> v42 cannot read it -> corrupts progressStore binary: old version not served, replacement needs review
Good Solution: stage the rollout so only a few are affected
Staged rollout: 1% -> watch -> advance1% × 50M = 500,000 with the bug, nothing to give thembounds the hurt, nothing for the already hurt - not enough
Great Solution: make the bad path switchable off from the server
Flag: server-side per install, every session; off in seconds, no download, no store, no player actionclient "rollback": refuse to enable, not removeScope: manifest revert for the self-delivered layer, never the store binarywhy: local migrations backward compatible one versionReview clock: binary fix hours to days, flag ~30 sStores do halt: Play staged rollout + halt, Apple phased release + pause; no download-timing control4 PB bill is the self-delivered layer'sCost: flags accumulate, pairs untested; expire at 100% for two releasesFlag cannot save: crash on launch, local data corrupt before the flag service -> canary + staged 1%
TRIGGER: "the client and server both changed" / "what deploys first"

2) Version skew, and the order of operations

SERVER FIRST, BACKWARD COMPATIBLE · EXPAND THEN CONTRACT · N-2 FOR 30 DAYS
Answer: server first, speaks both shapes; expand -> migrate -> contract; N-2 for 30 daysskew tested in CI; DB migrations one release ahead
Bad Solution: ship them together
Deploy both at once: server flips in minutes, clients take weekswindow: unmigrated clients vs a server without their protocol - outage for the majority
Good Solution: version the protocol and deploy the server first
Server speaks v1 + v2, clients migrate: right ordermissing: when the old path goes -> never, server accumulates every protocolmissing: skew untested until a player hits it
Great Solution: expand, migrate, contract, with the window written down
Expand: server accepts + writes both shapesMigrate: clients update; version distribution from the manifest serviceContract: old version below support floor -> old path deleted, ticketedPolicy: N-2 for 30 days; no number -> foreverSkew in CI: contract tests, new server × last N client protocols, every merge; fails a build, not a playerTax: every protocol change written three times, middle state live a monthDatabase, same recipe: expand schema -> dual-write -> backfill -> dual-read -> contractslower: bounded by data size, not adoptionafter a migration: server roll-forward onlymigrations ship >= one release ahead, never in the same deploy
TRIGGER: "50 million players need the patch" / "what does that cost"

3) The rollout is a bandwidth problem before it is a safety problem

4 PB · DOWNLOAD EARLY AND JITTERED · ACTIVATE ON A MANIFEST FLIP
Answer: 4 PB; fetch 48 h early, jittered; activate on a manifest flipstaged flip for risk, global flip for a content drop
Bad Solution: publish it and let everyone update
Peak:50M × 80 MB = 4 PB4 PB in 1 h = 1.1 TB/s = 8.9 Tbps - not buyable on demand, shapedpatch takes hoursBill: ~$80,000 either way
Good Solution: percentage cohorts
Cohorts: 1% -> 10% -> 50% -> 100%fixes the peak, same mechanism as safetycost: fairness, last cohort 2 days behind
Great Solution: decouple the download from the activation
Pre-download: 48 h early, background, jittered4 PB / 48 h = 23 GB/s - ordinary commitmentManifest flip, pick one:GLOBAL: all at once - content drop, fixes fairnessSTAGED: per cohort - risky release, costs fairnessclaiming both: contradictionTwo jobs: pre-download: bandwidth; staged activation: safetyDeltas vs chunking:chunk store primary: "give me the chunks I do not have", any version, no N-way matrixbinary diffs: common-hop optimisation, O(N) per release under N-2, tens of objectscost: diffing CPU on a 4 GB build, not storageContent-defined chunking: rolling hash boundariesfixed-size: 1-byte insertion shifts every later boundary, nothing dedupespack files chunk-aligned, per-chunk compressedContent-addressed: unchanged files never re-sent, one cache entry globallyNo pre-download: on demand at activation, small tailCost: two versions on client disk, up to 48 h
TRIGGER: "there are live matches on that server" / "how do you deploy without dropping people"

4) Deploying a stateful fleet without dropping sessions

DRAIN, DO NOT RESTART · A MATCH IS PINNED TO A VERSION · DEADLINE THE TAIL
Answer: drain with a deadline; match pinned to one version; capacity up firstdrain time bounded by match length
Bad Solution: rolling restart
Rolling restart: one instance at a timeweb request retries; 40-min match in memory does not10,000 instances -> outage every release
Good Solution: drain before replacing
Drain: no new matches, wait for existing to end - right mechanismTail: one long match or idle lobby blocks the instance foreverno deadline -> deploy never finishes, security patch stuck behind it
Great Solution: drain with a deadline, and pin the match to a version
Match pinned to one version: mixed versions, two rule sets - worse than a disconnectdefault, not physics: snapshot-and-migrate rejected on cost + live-state riskPersistent worlds, social hubs: no boundary-> scheduled shard downtime, shard transfer, or hot-reloadable script layerPast deadline, the RESULT: ranked match voided, entry refundedDrain time: bounded by match length; 40-min matches -> rollout in hours, budgetedTail: hard deadline + reconnect into new match or recovery flowCapacity leads the drain: spare new-version instances before old stop acceptingelse shortage at peakDeploy while players sleep, per region: free if fleet is regionalEmergency path: security fix -> accept dropping matches, decided in advance
TRIGGER: "who decides to halt" / "how do you know it is bad"

5) What stops a rollout, and how fast

CRASH-FREE SESSIONS · BAKE PER STAGE · AUTOMATIC HALT, HUMAN RESUME
Answer: crash-free session rate per version × platform, vs the old-version cohortbake per stage; halt automatic, resume manual
Bad Solution: watch the dashboards
Someone watches: 1% × 50M × 1/1000 crash = 500 players before anyone reactsnobody watches at 2am; 2am here, prime time elsewhere
Good Solution: alert on error rate
Page on threshold: fires at 2amwrong signal: launch crash never connects -> fewer server errorspage -> wake -> decide: minutes, rollout keeps advancing
Great Solution: gate on client-side health, and make halting automatic
Gate: crash-free session rate, per version × platformBaseline: old-version cohort, concurrent, never an absolute numberwhy: crash rate varies by day, region, deviceBake per stage: crash shows in minutes -> 20 min canary; leak / progression bug -> overnight canaryearliest stage bakes longestHalt automatic, resume manual: stopping safe, continuing riskyTwo services on every launch:manifest: fails OPEN to last-known-good; closed stops 200M installsflags: last cached value + short TTL + safe default per flagboth behind edge cache, not a hard dependency of launchClient-only metrics: crash-free sessions, launch success, download success, time-to-first-inputTelemetry broke: sudden DROP in reported sessions is the alarm; zero is not goodDetection window: launch crash reported on NEXT launch -> automatic halt in minutes60 s: halt propagation once decidedStats rule: bucket by device, OS, region; min sample + significance threshold

Four more dives, briefly

Signing: content addressing: integrity only vs a trusted manifest; forged manifest owns 200M installssign manifests + artifacts; client pins root key, verifies before apply; key in HSMprovenance per hash: {source, toolchain, dependencies}secrets fetched at run time by identity, never in an artifactPipeline trust: hermetic cached builds, same input -> same bytes; merge queue tests the comboflaky test quarantined same day, owner + deadline; pipeline p50 / p95 ownedretention: any chunk an N-2 client may need stays; GC by the 30-day policy, not disk pressureCI cost: 2,000 commits/day × 4 GB buildContent and config: code needs review; data, tuning, content packs do notbalance, events, drop rates ship as data in minutes; binary keeps the slow cadencedata breaks a client too -> same staged rollout, same health gatesMulti-platform: console + mobile stores review; PC + web do notsame release lands at different times; server compatible with allN-2 window sized to the slowest platformcert failure: schedule buffer, feature behind a flag either way
DONE WHEN: two or three dives opened; "I cannot un-ship a client" used as a REASON at least twice.
FLASHCARDS · THE FIVE HARDEST PROBESshow all (interview mode)
You shipped a client bug to the 10 percent cohort. What do you actually do?
Not a rollback: v43 on disk, pointer does nothingOrder: flag off (server-side, seconds) -> halt rollout, serve previous manifest -> hotfix forwardMobile: store review hours-days, flag 30 sFlag cannot save: crash on launch -> 1% cohort, need both
The client and the server both change the protocol. What order, and why?
Server first, backward compatibleExpand, migrate, contract: accept both -> migrate over weeks -> delete oldWindow: N-2 for 30 days, written; no number -> foreverContract tests in CI: new server × last N client protocolsTax: every change written 3×, middle state live 1 month
50 million players need an 80 MB patch. Do the math and tell me what breaks.
Total: 50M × 80 MB = 4 PB1 h: 1.1 TB/s - nobody sells48 h: 23 GB/s - ordinary CDNCost: 4 PB × $0.02/GB = ~$80,000 per rollout -> delta patchingFix: download != activationpublish 48 h early, jittered background fetch -> manifest flip, all at oncealso fixes cohort fairnessContent-addressed chunks: unchanged never re-sent; week-old client a small tail
How does a live match survive a server deploy?
Drain, not restart: stop routing new matches -> existing finish -> replaceOne server version per matchDrain time: longest match, not shutdown40 min matches -> fleet rollout hoursTail: hard deadline + reconnectwhy: one idle lobby blocks next security patchCapacity leads: new up before old stop acceptingSecurity fix: drop matches, decided in advance
What automatically stops a rollout, and how fast?
Primary gate: crash-free session rate per version × per platformNot server error rate: launch crash -> fewer server errorsCompare vs old-version cohort, not absolutewhy: varies by day, region, device; staged rollout = free control groupBake time: crashes minutes, leaks + progression hours -> early stages longerHalt: automatic, no approval. Resume: manual.Sudden drop in reported sessions: alarm, not silence
STEP 5 OF 5

Final design + what is expected at each level

wrap

Final Design

Commit to player: one pipeline, two very different last miles CI builds each commit once and publishes it to ArtifactStore by content hash; ReleaseBuilder reads those hashes and promotes the same bytes into a Release. The server image goes to the Deployer, which rolls it across the ServerFleet canary first while HealthGate can halt it and the Drainer lets live matches finish. The client bundle goes to PatchBuilder and the CDN, and the manifest to ManifestService, which GameClients ask before downloading chunks in the background. RolloutController flips the cohort, FlagService and KillSwitch turn a bad path off with no download, and a binary update waits on AppStoreReview. Commit to player: one pipeline, two very different last miles 2,000 commits a day. 10,000 server instances you control, 200M client installs you do not. BUILDbuild it once, hash it, and promote the same bytes forever Developer CI hermetic, cachedTierOne blocksmerge, under 10 min ArtifactStore content-addressed,immutablebuilt once, never rebuilt ReleaseBuilder picks the hashes,writes the manifest Release server image + clientbundle + manifestone version, promotednot rebuilt 1. Developer merges and CIbuilds the artifact once 2. CI publishes it under itscontent hash, and nothingrebuilds it again ReleaseBuilder reads thehashes, copies nothing 3. ReleaseBuilder promotes the samebytes into a Release what a Release pins, by hash and nothing else: { "version": "v43", "builtFrom": "commit 8f2a1c", "serverImage": "sha256:9c1e4b...", "clientBundle": "sha256:04bb7f...", "supersedes": "v42" } SERVER LAST MILEyou control the timing Deployer canary, then 5, 25,100 percent ServerFleet 10,000 instances HealthGate crash-free sessions,error rate, p99auto-halt in under 60 s Drainer no new matches,existing ones finish 4. Deployer rolls the server outcanary first, wave by wave 6. HealthGate haltsthe rollout the momenta gate breaks reports what thegates measure 5. Drainer lets livematches finish on theversion they started on what halts a wave, vs the cohort still on the old version: crash-free sessions more than 0.2 pp below iterror rate above 1.2x its ratep99 above 1.3x its p99 CLIENT LAST MILEthe player controls the timing ManifestService what should THIS install run?cohort, platform, channel PatchBuilder binary delta persupported version4 GB full becomes 80 MB GameClient background, resumable, jitteredv41, v42 and v43 all live CDN content-addressed chunks4 PB per full rollout 8. GameClient asks ManifestServicewhat it should be running the answer names the version and its chunks 7. PatchBuilder precomputes a deltafrom every version still supported 9. GameClient downloads chunks in thebackground, days before activation what ManifestService answers one install: { "version": "v43", "chunks": ["5d20...", ...], "activateAt": "2026-08-19T17:00Z" } // chunks: the v42-to-v43 delta, 80 MB CONTROLthe instant levers RolloutController halt, hold, advance FlagService server-side, per installthe only leverwith no download AppStoreReview hours to days, not ours KillSwitch off in seconds one flag set to off, no rollout needed how fast each lever really is: flag or kill switch: seconds, no downloadhalt or hold the cohort: under 60 sroll forward a client: hours to days the manifest for this version the client bundlefor this version the server imagefor this version 10. RolloutController flips the cohort, andactivation is instant for everyone in it 11. FlagService turns a bad path offwithout shipping anything at all a binary update waits here, a data-only patch does not Deploy the server first and backward compatible. Three client versions are live for weeks, so every protocol change is carried N-2.You cannot un-ship a client. Halting stops the spread, the flag stops the damage, and only a roll-forward actually fixes it.
1. CI: hermetic; tier one blocks merge, < 10 min at 2,000 commits/day2. ArtifactStore: content-addressed, immutable3. Release: {serverImage, clientBundle, manifest} by artifact id, never rebuilt4. Deployer: canary -> 5 -> 25 -> 100% over 10,000 instances5. Drainer: no new matches, live ones finish on their version6. HealthGate: crash-free, error rate, p99 -> auto-halt7. PatchBuilder + CDN: delta per supported version, 4 GB -> 80 MB; chunks by hash8. ManifestService: {cohort, platform, channel} -> version + chunks9. GameClient: background, 48 h early, jittered: 23 GB/s not 1.1 TB/s10. RolloutController: flips the cohort -> instant activation11. FlagService: server-side per session, only lever with no download"the server is a rollout problem and the client is a delivery problem"

What is Expected at Each Level

Mid:CI, artifact store, environments, canary deploystaged rollout + rollback plantreats client like server, misses N versions live at onceSenior:two last miles; server first, backward compatibledelta patching, staged cohorts, drain not restartflags as client-side safety; advance on healthtest tiers sized to commit rateStaff:4 PB unprompted: staging is a bandwidth requirementdownload split from activation: peak + fairness at onceno client rollback: halt, flag, roll forward; store review -> risky paths in configskew window: number + CI test; client-side health gate, telemetry drop is an alarmlimits: flags accumulate, two versions on disk, launch-crash caught by cohort only
DONE WHEN: point at the diagram, one breath: build once and promote, server first and backward compatible, download early and activate late, flags instead of rollback, halt automatically.