Design an end-to-end CI/CD pipeline for client updates (notes)

Functional Requirements

1. Merged commit -> production automatically, through gates.
2. Client patch -> staged rollout, background, resumable.
3. Bad release stoppable fast, server and client.
4. Old clients keep working mid-rollout.

Below the line (out of scope):

engine + asset build system (only need: deterministic artifact)cheat detection, telemetry schemas, localisation pipelinedeveloper environment tooling, local iterationStore review IN scope as constraint: hours to days -> risky behaviour ships as config, not binary

Non-Functional Requirements

Client bandwidth:50M daily × 80 MB patch = 4 PB egress per releasein 1 h: 1.1 TB/s - not buyableover 48 h: 23 GB/s - ordinary CDNstaged delivery: bandwidth requirement before safety mechanismCost:~$0.02/GB × 4 PB deltas = ~$80,000full 4 GB × 50M = 200 PB = ~$4Mdelta patching: budget lineall 200M installs: × 4Version skew:3+ client versions live at once, for weekssupport N-2 >= 30 days, every protocol change carried 3×Pipeline speed: PR feedback < 10 min, merge -> canary < 1 h500 engineers, 2,000 commits/dayClient patch: background, resumable, never blocks launch unless protocol brokeHalt: global <= ~60 s; bad code path off by flag, no downloadServer deploys: no dropped sessions; match pinned to one version -> drain time = match lengthArtifacts: immutable, content-addressed; tested bit-identical to shipped; promotion = pointer move

Defining the Core Entities

Artifact: {hash, kind: server image | client exe | pack file, bytes}immutable, name is the content hashRelease: {releaseId, version, serverImage, clientBundle, manifest}one version across all threeManifest: {version, platform, channel, cohort, chunks[] (content-addressed), activateAt}"flipping the manifest is the release"Channel: internal dogfood | alpha | beta | livestage before the 1% cohort, ~1 week on employeesCohort: {channel, bucket = hash(installId)}stable across launchesRollout: {releaseId, plan: [1, 10, 50, 100], step, bakeTime, healthGate, halted}FeatureFlag: {name, value, cohort}, server-side per install, no downloadthe client's rollbackDeployment: {releaseId, waves[], instance: drain state}

API or System Interface

GET    /v1/manifest?platform=&channel=&installId=&have=   // -> { version, chunks[], activateAt }
GET    /v1/flags?installId=&version=                     // -> { flagName: value }  // every session

POST   /v1/releases                     // { artifacts[] } -> promotes, never rebuilds
POST   /v1/rollouts                     // { releaseId, plan: [1,10,50,100] }
POST   /v1/rollouts/{id}/advance        // gated on bake time + health
POST   /v1/rollouts/{id}/halt           // global, under 60 s, no approval needed
POST   /v1/flags/{name}                 // { value, cohort } -> the instant lever

POST   /v1/deployments                  // server fleet: canary, then waves
GET    /v1/deployments/{id}/health      // crash-free sessions, error rate, p99

have=: server answers with a delta, 4 GB install -> 80 MB downloadactivateAt: download now, activate laterhint, not authority: client re-reads manifest at flip, server refuses new protocol before T-0why: cached manifest + halt, wrong clockmanifest edge cache TTL in seconds, bounds "haltable in 60 s"Flags: every session, never cached a dayin-flight match keeps bad path <= match length (~40 min)-> push revocations down the game socket; dangerous actions gated server-sideHalt: no approval. Advance: gated.POST /releases: artifact ids only, never a source ref, rebuild impossibleCohort: hash(installId), no 200M-row assignment tablereinstall -> new installId -> new cohort, acceptablesmall override table: QA, dogfood, support, allowlists

1) Commit to artifact: build once, then only promote

Test tiers: protect the 10-minute numbertier 1: unit + lint, < 10 min, blocks mergetier 2: integration + protocol contract, post-mergetier 3: soak + perf + full playthrough, nightly, files bugswhy: 2,000 commits/day × 40 min suite, queue never drainsBuild once, promote the same bytes:content-addressed, staging and prod same hashpromotion: pointer move, instant, reversibleRelease: server image + client bundle, one versionFlaky tests: quarantined same day

2) The server last mile: waves, gates, and draining a stateful fleet

Canary: 1 instance, real bake timeWaves: 5% -> 25% -> 100% of 10,000 instanceshealth gate each step: crash-free sessions, error rate, p99auto-haltDrain: stop routing new matches -> existing finish -> replace instancea match runs one server version end to enddrain time bounded by longest match, minutestail: hard deadline + reconnectHalt: stops spread, always safeRollback: second deploy, own drain, own risk

3) The client last mile: download now, activate later

Separate delivery from activation:publish patch 48 h early, background fetch, jitteredflip manifest -> clients with bytes activate4 PB / 48 h = 23 GB/s - buyable4 PB / 1 h = 1.1 TB/s - impossibleDelta patching: 4 GB client -> 80 MB downloadpatch builder precomputes diff per version in the fieldContent-addressed chunks: shared file downloaded once, CDN caches by contentBackground + resumable: drop at 79 of 80 MB, no restartForced update: only client < N-2, server no longer speaks it"we forced an update" is a retention eventClient details:Wi-Fi only by default, cellular opt-instorage-full path, two versions held during preloadapply time (console unpack + install)encrypt preload, key at unlock (dataminers)

4) Compatibility and the levers you actually have

Order: server first, backward compatible, alwaysExpand then contract, over weeks:server adds new field, still accepts oldclients migrate over the N-2 windowold version ages out -> server drops old fieldcontract tests in CI: new server × last N client protocolsSkew cost: 3 live versions × one server version per match = 3 matchmaking poolslonger queues, worse matches on smallest version; matchmaker version-awareFlags: the client's rollbackrisky code behind a flag by default; store review off critical pathLevers, in order: flip flag (seconds, no download) -> halt rollout -> hotfix forward (only real fix)

1) You cannot un-ship a client

Answer: flag off (seconds) -> halt rollout -> previous manifest -> hotfix forwardun-shippable: the store binary only

Launcher / asset layer: revert worksmanifest is the release, chunks content-addressed, old chunks on disk -> repoint to v42 (Steam: branch repoint)One-way door: local datav43 migrated save -> v42 cannot read it -> corrupts progressStore binary: old version not served, replacement needs review

Staged rollout: 1% -> watch -> advance1% × 50M = 500,000 with the bug, nothing to give thembounds the hurt, nothing for the already hurt - not enough

Flag: server-side per install, every session; off in seconds, no download, no store, no player actionclient "rollback": refuse to enable, not removeScope: manifest revert for the self-delivered layer, never the store binarywhy: local migrations backward compatible one versionReview clock: binary fix hours to days, flag ~30 sStores do halt: Play staged rollout + halt, Apple phased release + pause; no download-timing control4 PB bill is the self-delivered layer'sCost: flags accumulate, pairs untested; expire at 100% for two releasesFlag cannot save: crash on launch, local data corrupt before the flag service -> canary + staged 1%

2) Version skew, and the order of operations

Answer: server first, speaks both shapes; expand -> migrate -> contract; N-2 for 30 daysskew tested in CI; DB migrations one release ahead

Deploy both at once: server flips in minutes, clients take weekswindow: unmigrated clients vs a server without their protocol - outage for the majority

Server speaks v1 + v2, clients migrate: right ordermissing: when the old path goes -> never, server accumulates every protocolmissing: skew untested until a player hits it

Expand: server accepts + writes both shapesMigrate: clients update; version distribution from the manifest serviceContract: old version below support floor -> old path deleted, ticketedPolicy: N-2 for 30 days; no number -> foreverSkew in CI: contract tests, new server × last N client protocols, every merge; fails a build, not a playerTax: every protocol change written three times, middle state live a monthDatabase, same recipe: expand schema -> dual-write -> backfill -> dual-read -> contractslower: bounded by data size, not adoptionafter a migration: server roll-forward onlymigrations ship >= one release ahead, never in the same deploy

3) The rollout is a bandwidth problem before it is a safety problem

Answer: 4 PB; fetch 48 h early, jittered; activate on a manifest flipstaged flip for risk, global flip for a content drop

Peak:50M × 80 MB = 4 PB4 PB in 1 h = 1.1 TB/s = 8.9 Tbps - not buyable on demand, shapedpatch takes hoursBill: ~$80,000 either way

Cohorts: 1% -> 10% -> 50% -> 100%fixes the peak, same mechanism as safetycost: fairness, last cohort 2 days behind

Pre-download: 48 h early, background, jittered4 PB / 48 h = 23 GB/s - ordinary commitmentManifest flip, pick one:GLOBAL: all at once - content drop, fixes fairnessSTAGED: per cohort - risky release, costs fairnessclaiming both: contradictionTwo jobs: pre-download: bandwidth; staged activation: safetyDeltas vs chunking:chunk store primary: "give me the chunks I do not have", any version, no N-way matrixbinary diffs: common-hop optimisation, O(N) per release under N-2, tens of objectscost: diffing CPU on a 4 GB build, not storageContent-defined chunking: rolling hash boundariesfixed-size: 1-byte insertion shifts every later boundary, nothing dedupespack files chunk-aligned, per-chunk compressedContent-addressed: unchanged files never re-sent, one cache entry globallyNo pre-download: on demand at activation, small tailCost: two versions on client disk, up to 48 h

4) Deploying a stateful fleet without dropping sessions

Answer: drain with a deadline; match pinned to one version; capacity up firstdrain time bounded by match length

Rolling restart: one instance at a timeweb request retries; 40-min match in memory does not10,000 instances -> outage every release

Drain: no new matches, wait for existing to end - right mechanismTail: one long match or idle lobby blocks the instance foreverno deadline -> deploy never finishes, security patch stuck behind it

Match pinned to one version: mixed versions, two rule sets - worse than a disconnectdefault, not physics: snapshot-and-migrate rejected on cost + live-state riskPersistent worlds, social hubs: no boundary-> scheduled shard downtime, shard transfer, or hot-reloadable script layerPast deadline, the RESULT: ranked match voided, entry refundedDrain time: bounded by match length; 40-min matches -> rollout in hours, budgetedTail: hard deadline + reconnect into new match or recovery flowCapacity leads the drain: spare new-version instances before old stop acceptingelse shortage at peakDeploy while players sleep, per region: free if fleet is regionalEmergency path: security fix -> accept dropping matches, decided in advance

5) What stops a rollout, and how fast

Answer: crash-free session rate per version × platform, vs the old-version cohortbake per stage; halt automatic, resume manual

Someone watches: 1% × 50M × 1/1000 crash = 500 players before anyone reactsnobody watches at 2am; 2am here, prime time elsewhere

Page on threshold: fires at 2amwrong signal: launch crash never connects -> fewer server errorspage -> wake -> decide: minutes, rollout keeps advancing

Gate: crash-free session rate, per version × platformBaseline: old-version cohort, concurrent, never an absolute numberwhy: crash rate varies by day, region, deviceBake per stage: crash shows in minutes -> 20 min canary; leak / progression bug -> overnight canaryearliest stage bakes longestHalt automatic, resume manual: stopping safe, continuing riskyTwo services on every launch:manifest: fails OPEN to last-known-good; closed stops 200M installsflags: last cached value + short TTL + safe default per flagboth behind edge cache, not a hard dependency of launchClient-only metrics: crash-free sessions, launch success, download success, time-to-first-inputTelemetry broke: sudden DROP in reported sessions is the alarm; zero is not goodDetection window: launch crash reported on NEXT launch -> automatic halt in minutes60 s: halt propagation once decidedStats rule: bucket by device, OS, region; min sample + significance threshold

Four more dives, briefly

Signing: content addressing: integrity only vs a trusted manifest; forged manifest owns 200M installssign manifests + artifacts; client pins root key, verifies before apply; key in HSMprovenance per hash: {source, toolchain, dependencies}secrets fetched at run time by identity, never in an artifactPipeline trust: hermetic cached builds, same input -> same bytes; merge queue tests the comboflaky test quarantined same day, owner + deadline; pipeline p50 / p95 ownedretention: any chunk an N-2 client may need stays; GC by the 30-day policy, not disk pressureCI cost: 2,000 commits/day × 4 GB buildContent and config: code needs review; data, tuning, content packs do notbalance, events, drop rates ship as data in minutes; binary keeps the slow cadencedata breaks a client too -> same staged rollout, same health gatesMulti-platform: console + mobile stores review; PC + web do notsame release lands at different times; server compatible with allN-2 window sized to the slowest platformcert failure: schedule buffer, feature behind a flag either way

Not a rollback: v43 on disk, pointer does nothingOrder: flag off (server-side, seconds) -> halt rollout, serve previous manifest -> hotfix forwardMobile: store review hours-days, flag 30 sFlag cannot save: crash on launch -> 1% cohort, need both

Server first, backward compatibleExpand, migrate, contract: accept both -> migrate over weeks -> delete oldWindow: N-2 for 30 days, written; no number -> foreverContract tests in CI: new server × last N client protocolsTax: every change written 3×, middle state live 1 month

Total: 50M × 80 MB = 4 PB1 h: 1.1 TB/s - nobody sells48 h: 23 GB/s - ordinary CDNCost: 4 PB × $0.02/GB = ~$80,000 per rollout -> delta patchingFix: download != activationpublish 48 h early, jittered background fetch -> manifest flip, all at oncealso fixes cohort fairnessContent-addressed chunks: unchanged never re-sent; week-old client a small tail

Drain, not restart: stop routing new matches -> existing finish -> replaceOne server version per matchDrain time: longest match, not shutdown40 min matches -> fleet rollout hoursTail: hard deadline + reconnectwhy: one idle lobby blocks next security patchCapacity leads: new up before old stop acceptingSecurity fix: drop matches, decided in advance

Primary gate: crash-free session rate per version × per platformNot server error rate: launch crash -> fewer server errorsCompare vs old-version cohort, not absolutewhy: varies by day, region, device; staged rollout = free control groupBake time: crashes minutes, leaks + progression hours -> early stages longerHalt: automatic, no approval. Resume: manual.Sudden drop in reported sessions: alarm, not silence

Final Design

1. CI: hermetic; tier one blocks merge, < 10 min at 2,000 commits/day2. ArtifactStore: content-addressed, immutable3. Release: {serverImage, clientBundle, manifest} by artifact id, never rebuilt4. Deployer: canary -> 5 -> 25 -> 100% over 10,000 instances5. Drainer: no new matches, live ones finish on their version6. HealthGate: crash-free, error rate, p99 -> auto-halt7. PatchBuilder + CDN: delta per supported version, 4 GB -> 80 MB; chunks by hash8. ManifestService: {cohort, platform, channel} -> version + chunks9. GameClient: background, 48 h early, jittered: 23 GB/s not 1.1 TB/s10. RolloutController: flips the cohort -> instant activation11. FlagService: server-side per session, only lever with no download"the server is a rollout problem and the client is a delivery problem"

What is Expected at Each Level

Mid:CI, artifact store, environments, canary deploystaged rollout + rollback plantreats client like server, misses N versions live at onceSenior:two last miles; server first, backward compatibledelta patching, staged cohorts, drain not restartflags as client-side safety; advance on healthtest tiers sized to commit rateStaff:4 PB unprompted: staging is a bandwidth requirementdownload split from activation: peak + fairness at onceno client rollback: halt, flag, roll forward; store review -> risky paths in configskew window: number + CI test; client-side health gate, telemetry drop is an alarmlimits: flags accumulate, two versions on disk, launch-crash caught by cohort only