Design an end-to-end CI/CD pipeline for client updates

Functional Requirements

1. A merged commit should reach production servers automatically, through gates, without someone shepherding it.
2. A client patch should reach players as a staged rollout, downloaded in the background and resumable.
3. A bad release should be stoppable and its damage neutralisable fast, for both the server and the client.
4. Clients that have not updated yet should keep working while a rollout is in flight.

Below the line (out of scope):

• The engine and asset build system itself. I care that it produces a deterministic artifact, not how it compiles shaders.
• Cheat detection, telemetry schemas, and the localisation pipeline. They ride the same rails and would each eat the hour.
• Developer environment tooling and local iteration. Real, and a different design.
• Store review stays IN scope as a constraint, not as a workflow. It takes hours to days and it is the reason risky behaviour ships as configuration rather than as a binary. Cutting it entirely would remove the most interesting force on the design.

Non-Functional Requirements

• Do the client bandwidth number first, because it turns a safety idea into a hard requirement.
  • 50M daily players times an 80 MB patch is 4 PB of egress for one release.
  • Delivered in a single hour that is 1.1 TB a second, which is not a capacity anyone can buy on demand.
  • Spread over 48 hours it is 23 GB a second, which is an ordinary CDN commitment.
  • So staged delivery is a bandwidth requirement before it is a safety mechanism, and that is the sentence that makes the rollout design inevitable rather than optional.
  • It also costs real money, and the counterfactual is the number that lands: at roughly two cents a gigabyte, 4 PB of deltas is about 80,000 dollars, while shipping the full 4 GB build to the same 50M players would be 200 PB and about 4 million. That gap is why delta patching is a budget line and not an optimisation. Across all 200M installs rather than the daily actives, scale both by four.
• Then the version skew number, which is the other half.
  • Players update when they feel like it, so at any moment three or more client versions are live, for weeks.
  • Support N-2 for at least 30 days. That is a real tax: every protocol change is carried three times over.
• Pipeline speed: PR feedback under 10 minutes, merge to canary under an hour. 500 engineers and 2,000 commits a day means a slow pipeline is not an inconvenience, it is a queue that never drains.
• Client patch experience: background, resumable, and never blocking launch unless the protocol genuinely broke. A forced full-stop update is the worst thing you can do to a player who has 20 minutes to play.
• A rollout must be haltable globally within about 60 seconds, and a bad code path switchable off without any download at all.
• Server deploys must not drop live sessions. A match is stateful and pinned to one version, so drain time is bounded by match length rather than by process shutdown.
• Every artifact is immutable and content-addressed. What was tested must be bit-identical to what ships, which makes promotion a pointer move rather than a rebuild.

Defining the Core Entities

• Artifact is one built output addressed by the hash of its contents: a server image, a client executable, a pack file. Immutable by construction, because the name IS the content.
• Release bundles the artifacts that belong together: a server image, a client bundle, and the manifest that describes it. One version number spans all three, which is what makes "are these compatible" answerable.
• Manifest is the most important entity and the one people forget.
  • It is the answer to "what should THIS install be running", given platform, channel, cohort and current version.
  • It lists content-addressed chunks, so the client downloads only what it does not already have.
  • Flipping the manifest is the release. Everything before that is just moving bytes around, which is exactly why download and activation can be separated.
• Channel is the audience track: internal dogfood, alpha, beta, live. It is the cheapest safety mechanism available and it is the stage BEFORE the 1 percent cohort, so a release has usually been played by employees for a week before a stranger sees it.
• Cohort is a stable bucket of installs within a channel, derived by hashing the install id. Stable matters: a player must not oscillate between versions on every launch.
• Rollout is the state machine over cohorts: 1 percent, 10, 50, 100, with a bake time and a health gate between each step, and a halt that stops it dead.
• FeatureFlag is evaluated server-side per install and is the only lever that changes behaviour with no download at all. It is the client's rollback, and calling it that early sets up the whole dive.
• Deployment is the server-side sibling of Rollout: waves across the fleet, with drain state per instance.

API or System Interface

GET    /v1/manifest?platform=&channel=&installId=&have=   // -> { version, chunks[], activateAt }
GET    /v1/flags?installId=&version=                     // -> { flagName: value }  // every session

POST   /v1/releases                     // { artifacts[] } -> promotes, never rebuilds
POST   /v1/rollouts                     // { releaseId, plan: [1,10,50,100] }
POST   /v1/rollouts/{id}/advance        // gated on bake time + health
POST   /v1/rollouts/{id}/halt           // global, under 60 s, no approval needed
POST   /v1/flags/{name}                 // { value, cohort } -> the instant lever

POST   /v1/deployments                  // server fleet: canary, then waves
GET    /v1/deployments/{id}/health      // crash-free sessions, error rate, p99

• The manifest call carries what the client already haves, so the service answers with a delta rather than a full list. This is what turns a 4 GB install into an 80 MB download.
• activateAt is the separation of download from activation in one field. The client fetches now and switches later, which is the entire answer to the bandwidth problem.
• But a timestamp alone is a bug, and this is the case the halt has to survive. A client holding a cached manifest with a future activateAt will activate on its own clock even if the release was halted an hour ago, and a client with a wrong clock activates early.
• So activation is confirmed, not scheduled: the client re-reads the manifest at flip time and the SERVER refuses the new protocol before T-0. The timestamp is a hint for when to check, never the authority. Edge cache TTL on the manifest is then the real bound on "haltable in 60 seconds", so it is set in seconds, not minutes.
• Flags are fetched every session, not cached for a day. A kill switch that takes a day to propagate is not a kill switch.
• Per-session is still not "seconds" and the page should not pretend it is. A player already in a 40-minute match keeps the bad path for up to 40 minutes. So flag revocations are pushed down the live game socket, and anything genuinely dangerous is gated server-side per action rather than trusted to a client-cached value. Seconds for new sessions, session-length for in-flight ones, unless you push.
• Halt needs no approval and advance does. Stopping is always safe, so put no friction on it; proceeding is the risky direction, so gate that one.
• Creating a Release takes artifact ids, never a source ref. The API makes rebuilding impossible rather than merely discouraged, which is how the build-once rule survives contact with a deadline.
• Cohort comes from hashing the install id, not from a server-side assignment table. 200M rows of assignment is a database nobody needs, and a hash is stable across launches and updates.
  • Not across reinstalls: a reinstall usually mints a new install id, which is the point of one, so a player can land in a different cohort. Acceptable, and worth saying rather than claiming otherwise.
  • Keep a small override table anyway, for QA, dogfood, support escalations and allowlists. Pure hashing gives you no way to force one install into a cohort, and you will need that on day one.

1) Commit to artifact: build once, then only promote

• Test tiers exist to protect the 10-minute number, not to be thorough for its own sake.
  • Tier one, unit and lint, under 10 minutes, blocks the merge. It is the only thing that blocks a human.
  • Tier two, integration and protocol contract tests, runs after merge where it cannot stall a review.
  • Tier three, soak and performance and a full playthrough, runs nightly and files bugs rather than blocking a queue.
  • With 2,000 commits a day, a 40-minute blocking suite means the merge queue never drains, so this is arithmetic rather than preference.
• Build once and promote the same bytes, and make it structurally impossible to do otherwise.
  • A rebuilt binary is a different binary, so what you tested is not what you shipped.
  • Artifacts are content-addressed, so staging and production provably run the same hash.
  • Promotion is a pointer move, which also makes it instant and reversible on the server side.
• The release bundles the server image and the client bundle together under one version, which is what makes the compatibility question answerable later instead of guessed at.
• Flaky tests get quarantined the same day. The alternative is engineers learning that red does not mean broken, and once that is learned the gate is worth nothing.

2) The server last mile: waves, gates, and draining a stateful fleet

• Canary first, one instance, with a real bake time. A canary you advance past in 30 seconds has told you nothing except that the process starts.
• Then waves: 5 percent, 25 percent, 100 percent of 10,000 instances, with a health gate between each. The gate watches crash-free sessions, error rate and p99, and halts automatically rather than paging someone to notice.
• Draining is the part that is specific to this problem and the part worth spending time on.
  • A game server holds live matches, so you cannot restart it and you cannot move the match.
  • Draining means: stop routing NEW matches here, let existing ones finish, then replace the instance.
  • A match must run one server version end to end, so drain time is bounded by the longest match, not by process shutdown. That is minutes, and it has to be budgeted.
  • The tail needs a hard deadline with reconnect, or one very long match blocks the deploy forever.
• Halting is not rolling back, and the distinction matters under pressure. Halting stops the spread and is always safe. Rolling back is a second deploy with its own risk, its own drain, and its own chance of being wrong.

3) The client last mile: download now, activate later

• The central trick, and the thing I would say first: separate delivery from activation.
  • Publish the patch 48 hours early and let clients fetch it quietly in the background, jittered.
  • Flip the manifest at the release moment, and every client that already has the bytes activates instantly.
  • 4 PB spread over two days is 23 GB a second and buyable. The same 4 PB in the activation hour is 1.1 TB a second and impossible.
• Delta patching against every supported version. A 4 GB client that changed a little is an 80 MB download, and the patch builder precomputes a diff from each version still in the field rather than making the client reconstruct one.
• Content-addressed chunks mean a file shared between two releases is downloaded once, ever, and the CDN caches it globally by content rather than per release.
• Background and resumable is not a nicety. A player on mobile data who drops at 79 of 80 MB must not start again, and the download must never hold the game hostage while they are trying to play.
• Never block launch on a download unless the protocol genuinely broke, and be able to state exactly when that is true, because "we forced an update" is a retention event. The one honest case is a client older than N-2, which the server no longer speaks.
• The client-side details that decide whether players actually like this:
  • Wi-Fi only by default with an explicit opt-in for cellular, because a silent 80 MB on a metered plan is a complaint.
  • A storage-full path that fails gracefully and says so. Holding two versions during pre-download is exactly when disks fill, and this is the most common real update failure.
  • Apply time, not just download time. On console, unpacking and installing a patch is often longer than fetching it, and it is what the player actually waits through.
  • Encrypt the preload and ship the key at unlock, or dataminers read the content two days early. That is why pre-download in practice is encrypted rather than merely staged.

4) Compatibility and the levers you actually have

• Deploy the server first and backward compatible. Always. The reverse order breaks every client that has not updated, which is most of them on day one.
• Protocol changes follow expand then contract, over weeks:
  • The server adds the new field while still accepting the old one.
  • Clients migrate as they update, over the N-2 window.
  • Only after the old version ages out does the server drop the old field.
  • Contract tests in CI run the new server against the last N client protocol versions, so skew breakage is caught before merge and not in production.
• Skew has a user-visible cost the design has to own: the matchmaking pool splits. Three live client versions plus "a match runs one server version" means three pools, longer queues and worse matches for whoever is on the smallest version. The matchmaker is version-aware, not just the drainer, and that is the concrete price of the N-2 policy.
• Flags are the client's rollback, and this is the sentence the whole section exists for. Code already on a player's machine cannot be recalled, but the server can refuse to turn it on.
• So anything risky ships behind a flag by default, which also means the store review clock stops being on the critical path for fixing behaviour.
• Three levers, in the order you would actually reach for them: flip the flag (seconds, no download), halt the rollout (stops anyone else receiving it), ship a hotfix forward (the only real fix, and it takes as long as it takes).

1) You cannot un-ship a client

2) Version skew, and the order of operations

3) The rollout is a bandwidth problem before it is a safety problem

4) Deploying a stateful fleet without dropping sessions

5) What stops a rollout, and how fast

Four more dives, briefly

• Signing, which is the thing a CI/CD question is really about and which this design has so far left open.
  • Content addressing gives integrity only relative to a manifest you already trust. Whoever forges a manifest response owns 200M installs, which makes this the highest-value attack surface in the company.
  • So manifests and artifacts are signed, the client pins a root key and verifies before applying, and the signing key lives in an HSM rather than a CI environment variable.
  • Build provenance is the other half: record what source, what toolchain and what dependencies produced each hash, so a compromised dependency is traceable rather than theoretical.
  • Secrets never enter an artifact. They are fetched at run time by identity, which is also what lets the same immutable image run in staging and production.
• Making the pipeline itself trustworthy, which is what makes everything else possible.
  • Hermetic, cached builds so the same input gives the same bytes, which is also what makes content addressing meaningful.
  • Merge queues that test the combination rather than each PR alone, because two PRs that pass separately can break together.
  • Flaky tests quarantined the same day, with an owner and a deadline. A gate people have learned to ignore is worse than no gate, because it costs time and buys nothing.
  • Track pipeline p50 and p95 as a product metric owned by someone. It regresses silently, one minute at a time, until nobody can ship on a Friday.
  • Artifact retention is coupled to the support window, so it is not a housekeeping detail. Every chunk any N-2 client might still need has to stay alive, which means garbage collection is driven by the 30-day policy rather than by disk pressure, and deleting a chunk early is how you brick the clients that had not updated.
  • Price the CI side too. 2,000 commits a day against a 4 GB build is the other budget line, and it is what makes hermetic caching a cost decision as much as a correctness one.
• Content and configuration, which is the release valve for everything the store slows down.
  • Split what ships: executable code needs review; data, tuning, and content packs do not.
  • So balance changes, event schedules and drop rates go out as data in minutes, while the binary keeps its slow, safe cadence.
  • The catch to name: data can break a client just as thoroughly as code, so config gets the same staged rollout and the same health gates. Treating it as safe because it is "just data" is how you take down a client with a JSON file.
• Multi-platform, because the pipeline is not one pipeline.
  • Console and mobile stores have review; PC and web do not. So the same release lands on different platforms at different times, and the server has to be compatible with all of them at once.
  • That widens the skew window rather than changing its nature, and the N-2 policy has to be sized against the slowest platform, not the fastest.
  • Certification failure is a scheduling risk with no engineering fix, so the release plan carries a buffer and the feature is behind a flag either way.

Final Design

• CI builds hermetically and only tier one blocks a merge, so 2,000 commits a day still get feedback in under 10 minutes.
• Artifacts are content-addressed and immutable, a Release promotes the same bytes, and creating one takes artifact ids rather than a source ref so rebuilding is impossible rather than discouraged.
• The server rolls out canary then 5, 25 and 100 percent across 10,000 instances, with a health gate that halts automatically, and a drainer that lets live matches finish on the version they started on.
• The patch builder precomputes a delta from every supported version, and the CDN serves content-addressed chunks so an unchanged file is never sent twice.
• Clients fetch in the background 48 hours early and jittered, then the manifest flips and everyone in the cohort activates at once: 23 GB a second instead of 1.1 TB a second.
• The flag service is evaluated server-side every session and is the only lever that changes behaviour with no download at all.
• The invariant worth closing on: the server is a rollout problem and the client is a delivery problem, because I control one timeline and not the other. Server first and backward compatible, download early and activate late, and flags instead of rollback.

What is Expected at Each Level

• Mid
  • Builds a working pipeline: CI, artifact storage, environments, automated deploy with a canary.
  • Knows to stage a rollout and to have a rollback plan.
  • Usually treats the client like a server and assumes a rollback is available.
  • Usually misses that several client versions are live at once, so the compatibility question never comes up.
• Senior
  • Separates the two last miles and gets the deploy order right, server first and backward compatible.
  • Designs delta patching and staged cohorts, and handles draining a stateful fleet rather than restarting it.
  • Uses feature flags as the client-side safety mechanism and gates rollout advancement on health.
  • Sizes the test tiers against the commit rate instead of testing everything everywhere.
• Staff
  • Does the 4 PB arithmetic unprompted and concludes that staging is a bandwidth requirement, not only a safety one.
  • Separates download from activation, and can say why that specific move solves both the peak and the fairness problem at once.
  • Says plainly that a client rollback does not exist, and reframes the levers as halt, flag and roll forward.
  • Treats store review as a design force, pushing risky behaviour into config so the fast lever exists at all.
  • Writes the skew window down as a policy with a number, and tests skew in CI rather than discovering it in production.
  • Picks client-side health as the gate and knows that a drop in telemetry is an alarm rather than silence.
  • Names what the design does not do: flags accumulate and interact, two versions sit on client disk during pre-download, and a launch-crash bug is caught by the cohort rather than by any lever.