← all topics Realtime multiplayer notes · say it in this order 1 Problem2 Entities + API 3 High-level design4 Deep dives 5 Final + levelsreset

Design a real-time multiplayer system (Roblox, notes)

5M concurrent players · ~167K live sessions of up to 100 · simulate at 60 Hz, replicate at 20 Hz · p95 RTT (round-trip time) at or under 60 ms · 20 KB/s cap down per client, up to ~0.8 Tbit/s at peak · Hello Interview flow, board rules, one idea per line
Prompt as usually given: matchmaking service · regionally distributed game servers · edge servers for latency · client-server vs peer-to-peer (P2P "hard to moderate, cheat-prone"; Roblox: client-server + heavy latency optimisation) · three probes: millions of players, crash mid-session ("fallback to replication" or "migrate players"), cheating
SAY THIS SENTENCE FIRST

"The server owns anything that is scored or spent, and for those things a client only ever sends inputs."

Movement, the one real choice:server sim + client prediction: fair, server CPU per playerclient-owned physics: cheaper; speed + fly exploits (Roblox: client network ownership)default: server sim + prediction for competitiveclient ownership: casual-tier concession, bounds checksTriangle: latency / fairness / moderation, say where each is paid
STEP 1 OF 5

Understanding the problem

5 min

Functional Requirements

  1. Join: experience -> session near me, right people (fill | queue by skill or party)
  2. Play: inputs change a shared world, seen within ~200 ms, own actions instant
  3. Trust: no client asserts a hit, a score, or (competitive) a position
  4. Survive: crash loses <= 10 s of play + declared progress, never a purchase; back in ~10 s
Tier: per-experience flagcompetitive: server simulates movement, hot standbycasual: client-owned movement under bounds checks

Below the line (out of scope):

Game logic + scripting: inside the tick, creator's problemVoice chat, economy, marketplace: purchases only, in SurviveAsset delivery: ugc-assets; client has meshes + texturesModeration DECISION: ML problem; enforcement plumbing onlyAccount + auth: signed identity arrives with join

Non-Functional Requirements

Fleet: 5M concurrent, ~167K sessions, ~21K hosts:5,000,000 / 30 avg = ~167K sessions (max 100)one session = one process, nothing shared167,000 / 8 per host = ~21K hosts (8: design assumption, 12-16 cores)21,000 / 12 sites = ~1,750 hosts per regionone core per session, 16.7 ms tick; replication tick: 100 clients × ~30 interest decisionsfew hundred edge PoPs: terminate transport, no game state(Roblox: max players configurable, 700 announced; verify)
Tick: 60 Hz simulate, 20 Hz replicate:SIMULATE 60 Hz = 16.7 msREPLICATE 20 Hz = 50 ms = every third tick, delta vs last ACKed baseline(Roblox: ~20 Hz replication, 60 Hz Heartbeat, 240 Hz physics substep; verify the 240)INPUT 30 Hz: ~40 B + 2 × ~15 B + 28 B UDP/IP = ~100 B × 30 = ~3 KB/s upclient interpolates the gap
Bandwidth: ~14 KB/s landed vs 20 KB/s cap:Goal: 20 KB/s down per client (160 kbit/s), cellularUnit: ~40 B per entity: pos 6, rot 8, vel 6, anim+flags 4, id 2, + change mask + slack; 28 B UDP/IP header per PACKETNaive:100 × 40 B = 4 KB per client per update× 20 Hz = 80 KB/s per client× 100 clients = 8 MB/s (64 Mbit/s) per sessionO(N^2): 100 × 100 × 20 = 200,000 entity-updates/sInterest management:near: 30 × 40 B × 20 Hz = 24 KB/sfar:  70 × 40 B × 5 Hz  = 14 KB/s38 KB/s - over goal+ Deltas:38 × ~1/3 = ~13 KB/swhy: change masks, most fields unchanged in 50 ms+ 28 B × 20 Hz = 0.6 KB/s headers= ~14 KB/s - under cap, nicePer session: 100 × 160 kbit/s = 16 Mbit/sFleet:cap:    5M × 160 kbit/s = ~0.8 Tbit/slanded: 5M × 112 kbit/s = ~0.56 Tbit/s - KPI: replication bytes per client
Latency: p95 RTT <= 60 ms (assumption), hard cap ~150 ms, ~12 regional sites:OWN: 0 ms perceived, client predictionOTHERS: ~200 ms even at 50 ms RTTinput wait 0-33 ms+ one-way 25 ms+ next tick 0-17 ms+ next replication 0-50 ms+ one-way 25 ms+ interpolation buffer 2 × 50 ms = 100 ms= 150-250 ms, ~200100 ms buffer: a choice; 1 snapshot + extrapolate, jitter
Fairness: lag compensation cap 200 ms:server rewinds by shooter's one-way latency + interpolation delay200 ms RTT player = 100 + 100, at capbeyond cap: laggy player suffers (Valve Source default 1 s)
Matchmaking: two budgets:fill-join: p95 < 2 squeue-match (skill or party): seconds to ~1 min; this page: PLACEMENT only
Fault tolerance: loss bound:Detect: heartbeat to Fleet Manager every 1 s, 3 missed = dead, 3-5 sCheckpoint: delta 10 s, full 60 s, regional store, ~0.5-1 MB per full; engine state + declared staterestore: last full + <= 5 deltas; RESUME token TTL 60 s, back in ~10 sLoss bound: <= 10 s of play + DECLARED progress, never a purchase; undeclared: 60 s DataStore writeDurable: purchases: idempotent receipt, off tick, retried; progress: DataStore every 60 s + on leavewhy: DataStore per-server request budgets
Availability: join number + session KPI:region fails: joins elsewhere, sessions inside lost (checkpoint store regional)join: 99.9 percentsession: crash rate per session-hour, not nines
Triangle: latency, fairness, moderationprediction: spends fairness, buys latencyserver validation: spends latency, buys fairness + moderationplacement: buys latency, within a regionclient-owned physics: buys server CPU, costs fairness
DONE WHEN: "The server owns anything that is scored or spent, and for those things a client only ever sends inputs." + 60 Hz simulate, 20 Hz replicate, ~14 KB/s landed vs 20 KB/s cap.
STEP 2 OF 5

Entities + API

5 min

Defining the Core Entities

Player: {playerId, connection, inputSeq, resumeToken}, max one session at a timeExperience: {experienceId, maxPlayers <= 100, regionPolicy, mode: fill | queue(skill, party)}Session: {sessionId, experienceId, region, serverAddr, capacity, occupancy, state, tick}one session = one process, nothing sharedoccupancy reserved atomically in the Session Registry (Redis + Lua)GameServer: {region, capacity, health, lastHeartbeat}, heartbeat every 1 s to Fleet ManagerInput: {playerId, seq, clientTick, actions[]}"inputs are intents, not outcomes"Delta / Snapshot: {serverTick, ackSeq, baselineTick, changes[]}; snapshot = delta against nothingCheckpoint: {sessionId, tick, blob: engine state + declared state}, delta 10 s, full 60 s, ~0.5-1 MBRegion / EdgePoP: ~12 regions run game servers; few hundred PoPs terminate transport, no game state

API or System Interface

// control plane, HTTPS
POST /join {experienceId, playerId, party?, prefs?}
     -> 200 {sessionId, serverAddr, edgeAddr, joinToken}   // placed now (fill path); serverAddr is a routing handle, e.g. "euw1/s91"
     -> 202 {ticketId}                                       // queued (skill or party), then
GET  /join/{ticketId}                                        // push or poll until placed
POST /sessions/{sessionId}/leave

// internal, control plane
POST /fleet/allocate {experienceId, region}                  // -> a session on a warm server
PUT  /fleet/servers/{id}/heartbeat {load, sessions[]}       // every 1 s
POST /sessions/{id}/reserve                                  // atomic slot reservation in the registry

// data plane, a UDP-based reliable/unreliable channel protocol
C -> S  INPUT    {seq, clientTick, actions[]}               // 30 Hz, unreliable, carries the last 3 inputs, ~3 KB/s up
S -> C  DELTA    {serverTick, ackSeq, baselineTick, changes[]}  // 20 Hz, unreliable
S -> C  SNAPSHOT {serverTick, entities[]}                   // on join, or when the ACKed baseline is > 1 s old
S -> C  EVENT    {chat | kill | score | purchaseConfirmed}  // reliable channel
C -> S  RESUME   {sessionId, resumeToken}                   // on reconnect
Two planes:control: HTTPS; join, leave, allocate, heartbeatdata: UDP-based, 30 Hz up / 20 Hz down; retransmit worse than lossnative: UDP + thin reliability layer (Roblox: RakNet-derived)web: WebRTC data channels or WebTransportfallback: TCP 443; head-of-line blocking -> hitch at 20 Hzreliable channel: chat, kill, score, purchaseConfirmed; never positionJoin returns an address: client connects directly to game server via edgejoinToken: game server checks itedgeAddr: anycast to nearest PoP, then pinnedwhy: anycast route flaps break UDPserverAddr: opaque routing handle, <region>/<sessionId>, e.g. euw1/s91edge resolves it over backbone; client never learns a game-server IPsame handle on RESUME after crashJoin 200 vs 202: 200 fill-join, placed now; 202 queue ticket, push or pollnever retry POST /join in a loopINPUT: last 3 inputs redundant~40 B + 2 × ~15 B + 28 B header = ~100 B × 30 Hz = ~3 KB/s upno position field, intents onlyDELTA: ackSeq: inputs applied, client replays the rest; baselineTick: snapshot it is relative tobaseline > 1 s old -> full SNAPSHOTRESUME: data-plane, not a second jointoken TTL 60 s > detect 3-5 s + allocate + restoreregistry holds SEAT for the TTL; after 60 s, fresh join
DONE WHEN: "inputs are intents, not outcomes"; control plane HTTPS, data plane UDP + reliable channel for events only; ~3 KB/s up, 20 KB/s cap down landing ~14.
STEP 3 OF 5

High-level design

10 min, end to end, no dives yet

1) Join: matchmaking as placement

Join: matchmaking is placement, then the client talks to the server directly GameClient probes LatencyBeacons, then sends POST /join through the Gateway to the Matchmaker. The Matchmaker reserves one slot in SessionRegistry, or asks FleetManager to start a session on a warm process and register it first. The placement goes back through the Gateway, and GameClient then connects through the EdgePoP to the GameServer. Join: matchmaking is placement, then the client talks to the server directly pick the region from the player's own latency probe, reserve one slot atomically, hand back an address; the matchmaker is never on the hot path again GameClient probes LatencyBeacons on startknows its RTT per region Gateway Matchmaker region = best RTTpolicy: fill or queueparty, skill LatencyBeacon one per region FleetManager warm processes per regionbin-pack by loadallocate on empty SessionRegistry (Redis) (experienceId, region)-> sorted set by free slotsreserve = Lua check-and-decrementreservation TTL 30 s GameServers in eu-west SessionProcess 8 session processes per host~1,750 hosts here; ~21K over ~12 regionsone session = one process EdgePoP terminating proxy the numbers this shape has to survive: 5M concurrent players at peak~167K live sessions(avg 30, max 100)fill-join p95 < 2 sregion target: p95 RTT<= 60 ms (assumption) 0. GameClient probes LatencyBeacons on start RTT per region, geo-IP fallback 1. GameClient sends POST /join{experienceId, playerId, prefs} 2. Gateway forwards to Matchmaker 3. Matchmaker asks SessionRegistry for an open session in eu-west, reserves one slot atomic (Lua); if noneis open, step 4 4. Matchmaker asks FleetManager to allocate a session only when no opensession; pre-warmedprocess, registered,then reserved 4b. FleetManager starts the session on a warm process 4c. FleetManager registers the new session then step 3 reserves in it 5. Matchmaker returns {sessionId, serverAddr, edgeAddr, joinToken} back through the Gateway 5b. Gateway relays it to GameClient fill-join p95 < 2 s; skill/partyqueue is the game-matching topic(placement only here) 6. GameClient connects to the GameServer through the EdgePoP with the joinToken direct from here on; thematchmaker is out of the loop 6 (cont.) EdgePoP carries the flow over a pinned tunnel on the private backbone to the GameServer the edge terminates the transport, it does not simulate:no game state at the PoP, session state lives only on the GameServer the join wire (control plane, HTTPS) POST /join {"experienceId": "e42", "playerId": "p7", "prefs": {"mode": "fill"}}200 {"sessionId": "s91", "serverAddr": "euw1/s91", -- a routing handle, "edgeAddr": "edge.anycast:443", "joinToken": "eyJ..."} -- not a server IP202 {"ticketId": "t3"} when queued, then GET /join/t3 (push or poll)// the EdgePoP resolves serverAddr; the client never sees a game-server address reserve = one Lua script, check-and-decrement, atomic in Redis local free = redis.call('HGET', sess, 'free')if tonumber(free) < 1 then return 0 endredis.call('HINCRBY', sess, 'free', -1)redis.call('ZINCRBY', open, -1, sess) -- sorted set by free slotsredis.call('SET', 'rsv:'..token, sess, 'EX', 30) -- reservation TTLreturn 1 the not-so-happy paths: a reservation that is never used expires after 30 s and the slot returns to the session.the region has no capacity -> the next-best RTT region, and the player is told about it.allocation is the hot part in a launch (not the registry key): pre-warm the fleet, admit by ticket + push, never a retry storm.
Flow: client -> gateway -> Matchmaker -> region -> policy -> reserve slot -> {serverAddr, edgeAddr, joinToken} -> client -> edge PoP -> game serverRegion: client latency probe at app start, ~a dozen pings, cached; Geo-IP fallbackwhy: region decides RTTPolicy: fill | queue(skill, party)Fill: Session Registry, Redis keyed (experienceId, region), sorted set by free slotsreservation: Lua script, check + decrement atomic; TTL ~30 sjoin p95 < 2 s = registry read + Lua call + at worst one allocationEmpty: Fleet Manager, warm process (assets loaded), bin-pack fullest host with roomwhy: cold boot blows p95 past 2 sno capacity: next-best RTT regionAfter join: client -> edge PoP -> game server direct; matchmaker off the pathQueue: 202 + ticketId; match forms -> region minimising p95 RTT -> allocate -> reserve

2) Play: the authoritative loop, inputs in, deltas out

Play: inputs in, deltas out, the server is the only truth for anything scored or spent GameClient sends INPUT at 30 Hz to the GameServer, which ticks at 60 Hz, validates each input and applies it to WorldState. Every third tick the GameServer sends each client its own DELTA at 20 Hz. The client predicts its own inputs and reconciles on each DELTA; the server resolves hits by rewinding WorldState history. Play: inputs in, deltas out, the server is the only truth for anything scored or spent simulate at 60 Hz, replicate at 20 Hz; the client predicts itself and interpolates everyone else small dot = simulation tick, 60 Hz, one every 16.7 ms; big dot = replication tick, 20 Hz, every 3rd tick, a DELTA per client the server clock: t=0 replicate 16.7 33.3 50 replicate 66.7 83.3 100 replicate 116.7 133.3 150 replicate 166.7 183.3 ms GameClient predict own input immediatelykeep unacked inputson DELTA: rewind + replay(reconcile)render others 100 ms behind(interpolate 2 snapshots) GameServer authoritative: owns anythingscored or spent60 Hz simulate,20 Hz replicatevalidates every input GameClient the other players WorldState, tick N entities + a 200 ms historyring for lag-comp rewinds 1. GameClient sends INPUT {seq, clientTick, actions} at 30 Hz unreliable, carries the last 3 inputs so one lost packet loses nothing;~40 B newest + 2 x ~15 B delta-encoded older + 28 B header= ~100 B on the wire x 30 Hz = ~3 KB/s up tick 60 Hz read inputs ->validate (speed, cooldown,rate, ownership) ->simulate ->every 3rd tick: per-clientdelta 3. GameServer sends DELTA {serverTick, ackSeq, baselineTick, changes[]} at 20 Hz to GameClient delta vs the client's last ACKed baseline,prioritised by interest;<= 20 KB/s budget, lands ~14 KB/s 4. GameClient reconciles rewinds to the server state,replays unacked inputsafter ackSeq; own actionsfeel instant, disagreementsconverge here (smoothed;snap past a threshold) 3b. GameServer sends every other client its own DELTA (fan-out) per-client baseline and interest;worst case 100 x 20 KB/s per session 2. GameServer validates each input and applies it to the world bounds: max speed, cooldown,rate, ownership; done by theexperience's server script 5. GameServer resolves hits with lag compensation rewinds <= 200 ms to whatthe shooter saw; beyond thecap the laggy player suffers INPUT, client -> server (an intent, never an outcome) INPUT {"seq": 4127, "clientTick": 61230, "actions": [{"seq": 4125, ...}, {"seq": 4126, ...}, {"seq": 4127, "move": [0.7, 0.0], "jump": true}]}// 30 Hz, unreliable; the last 3 inputs ride along, so one lost packet// loses nothing. ~40 B newest + 2 x ~15 B delta-encoded older + 28 B header// = ~100 B on the wire x 30 Hz = ~3 KB/s up DELTA, server -> client (what changed since the baseline you ACKed) DELTA {"serverTick": 61233, "ackSeq": 4127, "baselineTick": 61230, "changes": [{"id": 17, "pos": [1203, 877, 41], "rot": 2891, "anim": 2}, {"id": 52, "pos": [988, 802, 41]}, ...]}// vs the client's last ACKed baseline; ~40 B per entity; not retransmitted,// the next DELTA covers a lost one. Full SNAPSHOT on join or baseline > 1 s old the numbers this shape has to survive: simulate 60 Hz (16.7 ms), replicate 20 Hz (50 ms)~40 B per replicated entity per updateown actions: 0 ms perceived (prediction)others seen ~200 ms after (150-250 at 50 ms RTT):sample 0-33 + 25 + tick 0-17 + rep 0-50 + 25 + interp 100 the not-so-happy paths: a lost DELTA is not retransmitted: the next delta covers it, because baselines are ACKed.the client's baseline is older than 1 s -> the server sends a full SNAPSHOT.an input fails validation -> clamped or dropped, and the client converges to the server's state on the next DELTA (smoothed).
Flow: inputs -> validate -> apply -> advance 16.7 ms -> every 3rd tick: delta vs ACKed baseline -> prioritise -> byte budget -> send -> client rewinds to ackSeq, replays unackedTick: 60 Hz, 16.7 ms; replication every 3rd tick = 20 HzValidate: max speed, cooldowns, input rate, ownership; fail -> reject or clampexperience's server script behind the seam (Roblox: server Luau)Simulation: one copy, this process; truth for anything scored or spentReplication: delta vs last ACKed baseline -> prioritise by interest -> cut to byte budgetunreliable channel; lost -> next delta covers itcap 20 KB/s, landed ~14 KB/sClient:prediction: own inputs applied same frame, un-ACKed inputs keptreconciliation: each DELTA -> rewind own entity to ackSeq -> replay unacked; disagree -> smooth, snap past thresholdinterpolation: others rendered between 2 newest snapshots, 100 ms behindHit registration: server rewinds targets to what shooter saw (one-way + 100 ms interp), cap 200 msMovement:server-simulated + prediction: fair, server CPU per player; competitive defaultclient-owned physics + bounds checks: cheaper (Roblox), speed/fly exploits; casual tierSecond player sees: own 0 ms; other ~200 ms at 50 ms RTTsample 0-33 + up 25 + tick 0-17 + replication 0-50 + down 25 + interp 100 = 150-250 ms

3) Latency: regions and edge

Latency: placement is the big lever, the edge terminates, it does not simulate The Matchmaker places the session in eu-west rather than us-east. The Lisbon GameClient connects to its nearest EdgePoP, which carries the flow over a pinned tunnel across the Backbone to the one GameServer; DELTAs come back the same way. A Sao Paulo GameClient reaches the same server through its own EdgePoP, and pays about 110 ms RTT for the distance. Latency: placement is the big lever, the edge terminates, it does not simulate a dozen regional server sites, a few hundred edge PoPs; state lives on exactly one server Matchmaker reads the players' latency mappicks the region for the session us-east region site another of the ~12; not this session GameServers SessionProcess GameClient in LisbonINPUT 30 Hz upDELTA 20 Hz down EdgePoP terminates the client's transportloss recovery on the short hoppinned unicast tunnel over the backboneno game state here Backbone the private networkPoP to region site eu-west region site Lisbon's RTT: ~8 + ~15 = ~23 ms,inside the p95 <= 60 ms target GameServer authoritative, one sessionthe only copy of the world60 Hz sim / 20 Hz repaddress never public GameClient in Sao Paulosame session, same server EdgePoP the nearest to them: also closeno game state here either not us-east: this pair's worst RTT would be higher there 1. Matchmaker places the session in eu-west, the regionminimising the players' p95 RTT: placement, the big lever tens of ms, more than any other lever 2. GameClient connects tothe nearest EdgePoP anycast for discovery,then pinned ~8 ms RTT the lossy last mile 3. EdgePoP carries the flow over apinned tunnel across the Backboneto the GameServer fewer congested public hops,loss recovery on the short first hop,address never public: a DDoS shield ~15 ms RTT backbone leg, private,predictable 3b. intothe site 4. GameServer sends DELTA 20 Hz back to GameClient same PoP, same tunnel; drawn direct for clarityyour own inputs feel instant (prediction); other players' actions land ~200 ms later (see the Play diagram) connects to theirnearest PoP, which isjust as close the first hop is shortfor everyone carries the flow over its own pinnedtunnel to the same server, but theBackbone cannot shorten the ocean ~110 ms RTT the distance itself: no PoP moves the server the edge cannot move the server; two players far apartshare one server and the far one pays the RTT under the ~150 ms playable cap, but about 5x the near player's RTT what each lever buys: placement: tens of ms, the biggest lever (~12 sites, p95 RTT <= 60 ms)edge: a few ms and stability, plus a DDoS shield (a few hundred PoPs)nothing fixes the distance between two players who chose to be far apartplayable hard cap ~150 ms RTT: the far player pays it, not the near one the not-so-happy paths: an EdgePoP dies: discovery re-pins the client to the next PoP, the session survives, its state was never there.the backbone link degrades: fall back to the public internet path, same server, worse jitter.a region fills: NEW sessions go to the next-best region; running sessions stay where they are.
Flow: input sample wait -> one way up -> next tick -> next replication -> one way down -> interpolation buffernetwork: only the two one-way legsLever 1, placement: region minimising p95 RTT of the session's players~12 regional sites; target p95 RTT <= 60 ms (assumption)Lever 2, edge: few hundred PoPs terminate client transport; pinned unicast tunnel over private backbone to regional serverloss recovery on the short first hop: a few msanycast: discovery + handshake only, then pin one PoPwhy: route flap mid-session breaks a UDP flowDDoS: client knows only {edgeAddr, session handle}Lisbon -> eu-west: ~8 ms last mile + ~15 ms backbone = ~23 ms RTT, under 60 ms targetbackbone down: public internet path, same server, worse jitter"edge terminates, does not simulate"PoP holds no game state; lose a PoP -> re-pin, RESUMEEdge does not fix: Sao Paulo on eu-west ~110 ms RTT, ~5× Lisbon, under 150 ms capsession is regional, cross-region play pays the RTTGlobal low latency: no; regional low latency + choose the region well

4) Survive: crash, checkpoint, rejoin

Survive: a crash costs at most 10 s of play and declared progress, never a purchase GameServer A checkpoints to CheckpointStore every 10 s and keeps purchases and progress in EconomyService and PlayerDataStore. When A stops heartbeating, FleetManager declares it dead, marks the session in SessionRegistry and allocates GameServer B, which restores the last checkpoint. GameClient checks the registry, then sends RESUME to B. Survive: a crash costs at most 10 s of play and declared progress, never a purchase heartbeat every 1 s, checkpoint every 10 s, dead after 3 missed, RESUME within ~10 s CheckpointStore regional object storeengine state + declared state, ~0.5-1 MBevery 10 s (delta), full every 60 slast: tick 41,100 GameServer A crashed at tick 41,300session 7f3a, this regionwas heartbeating every 1 s FleetManager expects a heartbeat every 1 sdead after 3 missed (~3-5 s)allocates the replacement GameServer B the replacement, a warm processrestores: last full + up to five deltasresumes at tick 41,100 durable state,never only in session memory nothing here was on GameServer A EconomyService purchases: idempotentreceipt,off the tick loop,retried until granted PlayerDataStore progress, inventory:every 60 s + on leave(DataStore request budgets) SessionRegistry handle -> host, statedead, then repointed to B GameClient socket dies -> 'reconnecting' UIRESUME {sessionId, resumeToken}token TTL 60 s 6. long before the crash, GameServer A had put everything durable outside itself purchases went through EconomyService as an idempotent receipt, off the tick loopprogress went to PlayerDataStore every 60 s and on leave; nothing durable was on A 1. GameServer A writes a checkpoint every 10 s, while itis alivethe last one istick 41,100 2. GameServer A stops heartbeating 3 missed (~3-5 s): declared dead 3b. FleetManager allocates GameServer B, a warm process 4. GameServer B restores from the last checkpoint loses <= 10 s of play: ticks41,100 -> 41,300 gone, 3.3 s 3a. FleetManager marks the session dead in SessionRegistry (then points it at B) seat held for the token TTL (60 s) 5a. GameClient asks the registry (via the Gateway) if the session is live again same handle; the edgeresolves it to B 5b. GameClient sends RESUME {sessionId, resumeToken} to GameServer B back in ~10 s end to end; the token outlives detect + allocate + restore the numbers this shape has to survive: loss bound <= 10 s of playand declared progressheartbeat every 1 s,dead at 3 missedcheckpoint every 10 s (delta),full every 60 sresume token TTL 60 s,seat held that longplayers back in ~10 s the RESUME wire RESUME {sessionId: "7f3a", resumeToken: "r9k2..."}-> same handle as before; the edge resolves it to GameServer B<- SNAPSHOT {serverTick, ...}: a full baseline, then DELTAs// token TTL 60 s > detect 3-5 s + allocate + restore the next ten seconds, for the fleet and for the player: t=-3.3 s last checkpoint, tick 41,100 t=0: crash tick 41,300; sockets die, clients show 'reconnecting' t=3-5 s: declared dead 3 missed heartbeats; FleetManager allocates B t=5-8 s: GameServer B up restored to tick 41,100 t=~10 s: players back RESUME accepted; loss <= 10 s of play the not-so-happy paths: the CheckpointStore itself is down: players return to matchmaking; up to 60 s of progress (the DataStore cadence) and the session are lost.competitive tier: state streamed to a shadow process each replication tick, failover in a tick or two, at double compute, so it is a per-experience tier, not the default.the resume token expires (60 s) with the player still away: the seat is released, a fill-join may take it, and the player comes back through a normal join.
Flow: heartbeat 1 s stops -> 3 missed, 3-5 s -> registry: session dead, seats held TTL 60 s -> warm replacement, same region -> restore last full + <= 5 deltas -> registry repointed, same handle -> client RESUME -> SNAPSHOT -> play on, ~10 sCheckpoint: regional store; delta every 10 s, full every 60 sfull: engine entity state + declared checkpointable; ~0.5-1 MB, 100-player worldLuau heap not serialisable -> declare-checkpointable API requiredingest ceiling: 167K × ~1 MB / 10 s = ~17 GB/s fleet, ~1.4 GB/s per regionfulls only: 1 MB / 60 s = ~2.8 GB/s fleet; a third at 30-player averagePlayer: socket dies -> "reconnecting", freeze, no lobby -> one registry read via gateway -> RESUME with token -> SNAPSHOTloss: <= 10 s of play + declared progress, never a purchasetoken TTL 60 s > detect + allocate + restoreDurable: purchase: idempotent receipt to economy service, off tick; progress: DataStore every 60 s + on leavewhy: DataStore per-server request budgetsStore itself dead: <= 60 s progress + the session lostConcede: not seamless; Roblox does not restore sessions (documented: DataStore + rejoin). Hot standby: a tier, double compute.
DONE WHEN: four paths; "edge terminates, does not simulate"; crash costs <= 10 s of play + declared progress, never a purchase.
STEP 4 OF 5

Potential deep dives

~20 min, interviewer steers
TRIGGER: "why not peer to peer" / "which sync model"

1) Client-server against peer-to-peer, and the sync model that follows

SERVER IS TRUTH · CLIENTS SEND INTENTS · PREDICT LOCALLY, RECONCILE ON DELTA
Answer: server owns anything scored or spent; clients send intentslatency: placement + prediction; smoothness: interpolation; fairness: lag compensation; bandwidth: deltas
Bad Solution: peer-to-peer, each client authoritative over itself
Model: each client simulates itself, tells others where it isSaving: simulation server only; each client still uploads to N-1 peers; STUN/TURN relays costLie: any client, any message; no moderation seamNetwork: NAT traversal unreliable; N(N-1)/2 links, 100 players = 4,950Legit P2P: deterministic lockstep + rollback (fighting, RTS); Luau on a physics solver is not deterministicPrompt: P2P "hard to moderate, cheat-prone"
Good Solution: authoritative server, full snapshots, clients as dumb terminals
Model: one server owns the world; inputs in, full state out at fixed rate; client draws last receivedWins: fair + moderatable, one place decides every inputCosts:every action waits a round trip: 60 ms RTT -> jump >= 60 ms after press, ~100 with tick + replication waitsO(N^2): 100 × 100 × 20 Hz = 200,000 entity-updates/s, 8 MB/s per session
Great Solution: authoritative server, plus prediction, interpolation, deltas and lag compensation
Server: only truth for anything scored or spent, every input validated; client hides the round tripPrediction, own entity: input applied same frame, unacked kept; each DELTA: rewind to ackSeq, replay; disagree -> smooth, snap past thresholdInterpolation, others: between 2 newest snapshots, 100 ms behind; ~200 ms to see another act at 50 ms RTTLag compensation, server: rewind targets to what shooter saw (one-way + interp delay), cap 200 msDeltas, wire: vs last ACKed baseline; 40 B per entity quantised; ~1/3 bytes -> 20 Hz affordable; bandwidth lever, not latencyException: Roblox hands physics ownership of own character to the client -> speed + fly exploits -> Hyperion (Byfron, 2023) client anti-cheat (verify)server sim + prediction for competitive; client ownership only non-competitive physics or casual tier, never scoredTriangle: prediction spends a little fairness to buy latency"the server never adopts a predicted state, it only lets the client draw one"
TRIGGER: "100 players in a session, what does the network look like"

2) 100 players in one session: what the network looks like

FULL STATE IS 8 MB/S A SESSION · INTEREST GETS TO 38 KB/S · DELTAS LAND IT AT ~14 KB/S UNDER A 20 KB/S CAP
Answer: naive 80 KB/s per client, 8 MB/s per session, O(N^2); interest 38 KB/s; + deltas ~14 KB/s vs 20 KB/s capper session 16 Mbit/s; fleet 5M × 160 kbit/s = ~0.8 Tbit/s at cap, ~half at landed
Bad Solution: full state broadcast every replication
Model: every 50 ms, every client gets the whole worldMath:100 entities × 40 B = 4 KB per client per update× 20 Hz = 80 KB/s per client× 100 clients = 8 MB/s (64 Mbit/s) per sessionO(N^2): 100 × 100 × 20 = 200,000 entity-updates/s; 2× players -> 4×Cost: 80 KB/s = ~290 MB/hour of a phone's plan, ~6× egress of landed 14 KB/sserver: 64 Mbit/s per process, 512 Mbit/s per 8-process host
Good Solution: deltas against the last ACKed baseline
Model: changes since client's last ACKed snapshot, per-field change masksSaving: ~1/3 of full state: ~25-30 KB/s per client - overQuantisation: already in the 40 B: pos 6, rot 8, vel 6, anim + flags 4, id 2, + change mask + slack28 B UDP/IP header per PACKET, not per entityStill O(N^2): every client hears every entity; smaller constant, same shape
Great Solution: interest management, a per-client priority budget, and ACKed baselines
Interest management: ~30 near at full rate, far at 5 Hz, rest not at all (Roblox: StreamingEnabled)Math:interest alone: 24 + 14 = 38 KB/s - overdeltas alone: ~25-30 KB/s - overboth: 38 × ~1/3 = ~13 + 0.6 headers = ~14 KB/s vs 20 capPriority accumulator + byte budget per update: cap is a guarantee, not an averageACKed baselines: lost delta covered by the next; baseline > 1 s old -> full SNAPSHOTCost: CPU per client per update -> alert on tick > 16.7 ms
TRIGGER: "how do you prevent cheating"

3) Cheating, and why prediction is the crack it lives in

VALIDATE EVERY INPUT · HITS DECIDED ON THE SERVER · LAG COMP CAPPED AT 200 MS
Answer: server validates every input vs bounds, decides every hit, lag comp cap 200 ms"the server never adopts a predicted state; it only lets the client render one"
Bad Solution: trust the client
Client says: "I am here", "I hit him"; server writes it downspeed hacks, teleports, aimbots, infinite score: one modified client awayP2P failure mode inside client-server
Good Solution: the server validates every input against bounds
Inputs vs bounds: max speed, cooldowns, rate limits, target ownership -> reject or clampsame tick as the input, no extra round tripNot covered: hits under latency; inhuman but inside every bound
Great Solution: server-side hits with bounded lag compensation, the platform seam around untrusted developer code, physics ownership only for unscored objects, offline behavioural anti-cheat
Hits server-side, lag comp cap 200 ms:rewind target by one-way latency + interpolation delay, test, returncap: fairness choice (Valve Source default 1 s)200 ms RTT (100 + 100): at cap; 400 ms: cannot hit a target that moved 300 ms agoWho validates: the experience's SERVER script (untrusted Luau) behind the platform seamseam: server scripts + RemoteEvents, only path to server statewalls: Luau sandbox (script timeouts, memory limits); process isolation per instance (our design); per-connection rate limitsclient-side anti-cheat: Hyperion (Byfron, 2023), publicly documentedwhy: character physics client-ownedINPUT schema: {playerId, seq, clientTick, actions[]}, no position field to lie inEdge: 300 inputs/s -> throttled; no session token -> never reaches simulation; (DDoS shield)Physics ownership only for unscored objects: barrel yes; character, projectile, score nevercasual tier concedes character ownership: where exploits areOffline behavioural anti-cheat: logged inputs + outcomes -> outlier pipeline (accuracy, reaction time), replay; ban or shadow flag, not in the tickModeration seam: chat, actions, events pass the server -> logged, filtered, enforcedTension: prediction is the crack; client renders unconfirmed state, server never adopts itconverge: small corrections smoothed, past error threshold snappedscored outcomes: server, validated inputs, server time
TRIGGER: "server crashes mid-game"

4) The server crashes mid-game

HEARTBEAT 1 S, DEAD AT 3 MISSED · CHECKPOINT DELTA 10 S, FULL 60 S · RESUME IN ~10 S, LOSE AT MOST 10 S
Answer: heartbeat 1 s, dead at 3 missed; checkpoint delta 10 s + full 60 s; restore on warm process, RESUME; loss <= 10 s, never a purchasehot standby fed each tick: tick or two of loss, double compute, per-experience tier
Bad Solution: session gone, everyone back to the lobby
Process dies, sockets close: 100 players -> matchmakingprogress in session memory: gone; purchase applied before recorded: paid for nothingdifferent session, different people: competitive round lost
Good Solution: heartbeat, checkpoint, restore, RESUME (the HLD)
Fleet Manager: dead in 3-5 s; restore last full + <= 5 deltas (<= 10 s old) on warm process, same region; clients RESUME with tokensLoss bound: <= 10 s of play + declared progress, never a purchase; back in ~10 spurchases: idempotent receipt, off tick; progress: DataStore every 60 s + on leavePlayer sees: "reconnecting" ~10 s, same world, same people, rewoundGap: visible interruption; ranked 10 s rewind: real complaint
Great Solution: a hot standby as a tier, the client side, and the store-that-died case
Checkpoint: delta 10 s, full 60 s, ~0.5-1 MB, engine + declared state; restore = last full + <= 5 deltasLuau heap not serialisable; Roblox does not restore sessions; 17 GB/s ingest ceiling at 100-player worldsCompetitive tier, hot standby by STATE STREAMING:input-log replay needs deterministic sim; Luau + physics solver + os.time + HttpService: notshadow process fed each replication tick; failover = tick or two; clients RESUME to shadowCost: double compute for a rare event -> per-experience tier, not defaultfleet KPI: crash rate per session-hourClient timeline:t=0:      socket dies, "reconnecting", last state frozent=3-10 s: retry registry read via gateway, handle unchanged~t=10 s:  RESUME with token -> full SNAPSHOT -> play on, rewound <= 10 stoken TTL 60 s > detect (3-5 s) + allocate + restore; seat held for TTL; after 60 s fresh joinFleet timeline:t=0:      crasht=1-3 s:  heartbeats missedt=3-5 s:  dead; marked dead in registry; seats held for token TTLt=5-8 s:  replacement on warm host; last full + <= 5 deltas restoredt=8-10 s: registry repointed, live; clients RESUME, same handle; loss <= 10 sPurchases vs progress:purchase: idempotent receipt (ProcessReceipt-style), off tick, retried; no double-grantprogress: DataStore every 60 s + on leave, key events batched; checkpoint covers the gapwhy: DataStore per-server request budgetsCheckpoint store itself died: back to matchmaking; lost: <= 60 s progress + the session; purchases safeWhole-region failure: new joins elsewhere via latency map; sessions inside lost, store is regionalunless cross-region store replication or cross-region standby99.9 percent join target: join can move, a session cannot
TRIGGER: "millions of players"

5) Millions of players: what is sharded by what, and where it still gets hot

THE SESSION IS THE SHARD · REGISTRY BY (EXPERIENCE, REGION) · IN A LAUNCH, CAPACITY IS THE HOT PART
Answer: "the session is the shard"; registry by (experienceId, region); launch: capacity spike, not registry spike1M joins / 600 s = ~1,667/s trivial; 33K sessions / 8 per host = ~4,100 hosts vs ~1,750-host region -> pre-warm from launch calendar
Bad Solution: one matchmaker, one registry, scale by adding game servers
Add hosts, one matchmaker + one registry:game servers fine; control plane: every join through one placement pathlaunch: no open sessions -> empty path; Fleet Manager cold-boots -> falls behind; clients retry POST /join -> retry storm
Good Solution: shard the control plane the same way the fleet is already sharded
Session = unit of scale: one per process, nothing shared; 167K sessions on 21K hostsControl plane sharded the same way:Matchmaker by (experienceId, region)Session Registry: Redis partitioned by the same keyFleet Manager per region: bin-packing, pre-warmed processes, autoscale on occupancyNot enough for launch: capacity spike inside ONE region of ~1,750 hosts
Great Solution: pre-warm on the calendar, size host capacity and spin-up rate, admit through a queue, and keep cross-session traffic off the loop
Hot LAUNCH: 1M players, one experience, 10 minutesjoins: 1,000,000 / 600 s = ~1,667/s fleet-wide; ~140/s per region key; ~800/s if one region takes halfRedis sorted set + Lua reservation: trivial - not the bottlenecksessions: maxPlayers 100 -> 10K (~17 starts/s); maxPlayers 30 -> 33K (33,000 / 600 = ~55/s)capacity: 33K / 8 per host = ~4,100 hosts vs one region of ~1,750 -> more than double(~20 percent of the 21K fleet only if spread evenly)one core per session -> real hardware - THE hot partPre-warm from launch calendar: hosts provisioned, warm asset-loaded processes registered before first joinspin-up rate ahead of 17-55 sessions/s - secondary knobQueue when allocation lags: join returns 202 + ticket, push on allocation; never a retry stormSalt the hot key: sorted set + Lua spread across Redis partitions; cost: fill quality - an extra, not the fixCross-session features over a message bus: publish and move on; nothing off-session on the 16.7 ms tick pathRoblox: MessagingService (pub/sub, rate-limited), MemoryStoreService (ephemeral shared state), TeleportService + ReserveServer, DataStoreService (durable, per-server budgets)Still hot, and the bill:Fleet Manager for the launch region: one process per region, queue in frontfleet ~21K hosts (one core per session, 8 per 12-16 core box, design assumption)egress ~0.8 Tbit/s at cap, ~0.56 expected at landed 14 KB/s -> replication bytes per client is the KPI

Four more dives, briefly

Transport: UDP + thin reliability layer (Roblox: RakNet-derived); unreliable: INPUT, DELTA; reliable: EVENTweb: WebRTC data channels or WebTransport; fallback TCP 443 costs head-of-line blockingINPUT carries last 3 inputs: ~40 + 2 × ~15 + 28 B header = ~100 B, ~3 KB/s -> single drop loses nothingTick rate: 60 Hz sim = 16.7 ms; 20 Hz replicate (Roblox: ~20 Hz replication, 60 Hz Heartbeat, 240 Hz physics substep, verify the 240)inputs 30 Hz; 60 Hz: ~8 ms less input wait for ~6 KB/s up - competitive tierreplication 40 Hz: ~28 KB/s -> priority budget clips, "the cap is a cap"; sim 120 Hz halves CPU headroomnot deterministic across machines (Luau, physics solver, os.time) -> standby streams state, no lockstepClock sync: client clock ahead by one-way latency + margin; server tick + RTT/2 from handshake, updated from DELTA serverTickserver jitter buffer 1-2 ticks; early or late beyond it: dropped (late = speed-hack vector)0-33 ms input sample wait at 30 HzObservability: alert on "tick over budget", not "CPU high"per session: tick p50/p99 vs 16.7 ms; bytes per client vs 20 KB/s (~14 expected); inputs rejected/s, dropped lateper region: RTT p95 <= 60 ms; heartbeat misses; crash rate per session-hour; checkpoint write latency; join p95 vs 2 shost 40 percent CPU, one session at 25 ms: stutter for 100 people
DONE WHEN: 2-3 dives opened; "the server never adopts a predicted state" said once; 8 MB/s -> ~14 KB/s arithmetic shown.
FLASHCARDS · THE FIVE HARDEST PROBESshow all (interview mode)
Why client-server rather than peer-to-peer, given P2P is lower latency and cheaper?
P2P: any client can lie, no seam every message passes throughno moderation, no hit or score validationscored or spent -> server owns itAuthority costs latency, bought back:placement: p95 RTT <= 60 msprediction: own entity, 0 ms perceivedinterpolation: others, 100 ms bufferlag compensation: cap 200 msanother player's action still ~200 ms late at 50 ms RTT"Cheaper", "faster":P2P saves the simulation bill, not client bandwidth (N-1 uploads; STUN/TURN still cost)N(N-1)/2 = 4,950 connections for 100 + NAT traversal - not reliably lowerLegit P2P: deterministic lockstep + rollback (fighting games, RTS); Luau on a physics solver is not deterministicRoblox exception: physics ownership of own character -> client (documented)why: speed / fly exploits; client anti-cheat (Hyperion, Byfron 2023)mine: server sim + prediction for competitive tier; bounds validation alwaysclient ownership: unscored, or casual tier, conceded out loudTriangle: latency, fairness, moderationP2P pays fairness + moderation for latencyclient-server pays latency, buys most of it back
A player with 200 ms RTT shoots at someone. Who decides whether it hit, and as of when?
Who: server, always (Roblox: experience's server script)client sends {fire, seq, clientTick}, never "I hit him"As of when: when the shooter SAW the targetrewind others by one-way latency + 100 ms interpolation delay, test, back to presentLag compensation, cap 200 ms:this shooter: 100 + 100 = 200 ms, at cap, compensated in full300 ms RTT: extra not compensateda choice, not a constant (Valve Source 1 s)why cap: else laggy attacker shoots into the past, victim shot around a cornerVictim's client never decides: hit = EVENT on reliable channel; health is server statePrediction: muzzle flash instant, damage only when server says"the server never adopts a predicted state, it only lets the client render one"Bounds first: cooldown -> dropped; rate limit -> throttled; then hit test
100 players in one session: what is the bandwidth, and how do you get it under 20 KB/s per client?
Unit: ~40 B per entity per update, quantised, 20 Hzpos 6, rot 8, vel 6, anim+flags 4, id 2, + change mask + slack28 B UDP/IP header per PACKETNaive:100 × 40 B × 20 Hz = 80 KB/s per client× 100 clients = 8 MB/s (64 Mbit/s) per sessionO(N^2): 200,000 entity-updates/sInterest only (Roblox: StreamingEnabled):30 × 40 B × 20 Hz = 24 KB/s70 × 40 B × 5 Hz  = 14 KB/s38 KB/s - over+ Deltas:38 × ~1/3 = ~13 KB/swhy: change masks, most fields unchanged in 50 ms+ 28 B × 20 Hz = ~0.6 KB/s headers~14 KB/s vs 20 KB/s (160 kbit/s) cap - under, both levers neededPriority accumulator + byte budget per update: cap is a guarantee, not an averageACKed baselines: lost delta covered by the next; baseline > 1 s -> full SNAPSHOTPer session: 100 × 160 kbit/s = 16 Mbit/s worst caseFleet: 5M × 160 kbit/s = ~0.8 Tbit/s cap; ~0.56 Tbit/s at landed 14 - KPIUpstream: ~100 B × 30 Hz = ~3 KB/s per clientCost: CPU per client per update -> alert on tick > 16.7 ms; egress is the bill
The game server crashes mid-session. Walk the next 30 seconds for the fleet and for the player.
Fleet:0-3 s:  heartbeats stop3-5 s:  3 missed -> dead; session marked dead in registry; seats held for token TTL5-10 s: new process on warm host; restore last full + <= 5 deltas; registry repointed, same handlePlayer:0:     "reconnecting", freeze on last state~10 s: RESUME with token (TTL 60 s) -> SNAPSHOT -> play onloss:  <= 10 s of play + declared progress, never a purchaseDurable: purchases = idempotent receipt, off tick; progress = DataStore every 60 s + on leaveStore itself dead: back to matchmaking, <= 60 s progress lostConcede: not seamless; Roblox does not restore sessions. Hot standby: a tier, double compute.
Millions of players at once: what is sharded by what, and where does it still get hot?
"the session is the shard"5M / 30 per session = ~167K sessionsone process, one core; 8 per host (design assumption, 12-16 core)~21K hosts, ~12 regionsnothing shared -> horizontal, no coordinationControl plane:Matchmaker: sharded by (experienceId, region)Session Registry: Redis, same partitionFleet Manager: per region; bin-packing, pre-warm, autoscale on occupancyLaunch: 1M players, one experience, 10 min = ~1,667 joins/s~140/s per region key spread; ~800/s if one region takes halfregistry NOT the bottleneckHot: regional CAPACITY:maxPlayers 100: 10K sessions (~17 starts/s)maxPlayers 30:  33K sessions (~55 starts/s)33K / 8 = ~4,100 hosts vs ~1,750-host launch region - more than doubleanswer: pre-warm from launch calendar; spin-up rate second; ticket queue with push, never clients retrying POST /joinCross-session: message bus, never the session loopMessagingService, MemoryStoreService, TeleportService + ReserveServer, DataStoreService (per-server request budgets)nothing outside the session on the 16.7 ms tick pathHot by design: Fleet Manager, launch region; one per region, queue in frontBill: ~21K hosts; egress ~0.8 Tbit/s cap, ~0.56 expected -> replication bytes per client is the KPI
STEP 5 OF 5

Final design + what is expected at each level

wrap

Final Design

Real-time multiplayer: the whole system On the control plane, a GameClient joins through the Gateway, the Matchmaker reserves a slot in SessionRegistry or asks the regional FleetManager to allocate a session, and the placement comes back to the client. On the data plane the client talks through an EdgePoP and the private Backbone to its GameServer. GameServers heartbeat FleetManager, checkpoint to CheckpointStore, write progress to PlayerDataStore, take purchases through EconomyService, stream to TelemetryPipeline and trade cross-session events over MessageBus, so nothing durable lives only on the game server. Real-time multiplayer: the whole system control plane places, data plane plays, everything durable lives off the game server Gateway HTTPS control planePOST /join, /leaveauth, rate limit Matchmaker region = best RTT from the probepolicy: fill or queue, party, skillplacement only,never on the hot path SessionRegistry Redis, keyed(experienceId, region)sorted set by free slotsreserve = Luacheck-and-decrement, TTL 30 s 3. Gateway forwards thejoin to Matchmaker 5a. Matchmaker returnsthe placement to Gateway 4. Matchmakerreserves oneslot inSessionRegistry atomic; if nosession is open, 4b GameClient native + web, millionspredicts itself,interpolates otherskeeps a resumeToken LatencyBeacon one per regionpinged on app start EdgePoP terminating proxy,no game stateanycast to find, then pinnedloss recovery onthe short hop Backbone private, pinnedunicast tunnelserver address never public 2. GameClient sends POST /join{experienceId, playerId, prefs}to Gateway 5b. Gateway returns {sessionId,serverAddr, edgeAddr, joinToken}to GameClient fill-join p95 < 2 s; skill/party queueis the game-matching topic 1. GameClient probesLatencyBeacons on start RTT per region, geo-IP fallback 6. GameClient connectsthrough EdgePoP overthe Backbone to itsGameServer INPUT 30 Hz up(~3 KB/s), DELTA20 Hz down (<= 20 KB/s);the edge terminates,it does not simulate the data plane wire (UDP) INPUT {seq, clientTick, actions[]} 30 Hz up, unreliable, carries the last 3DELTA {serverTick, ackSeq, baselineTick, changes[]} 20 Hz down, vs the ACKed baselineSNAPSHOT (full) on join, or when the baseline is > 1 s oldEVENT {chat, kill, score, purchase confirmed} reliable channelRESUME {sessionId, resumeToken} on reconnect, token TTL 60 stransport: UDP + thin reliability layer; WebRTC orWebTransport on web; TCP/443 fallback the control plane wire (HTTPS) POST /join {experienceId, playerId, party?, prefs?} 200 {sessionId, serverAddr, edgeAddr, joinToken} placed: connect directly, p95 < 2 s 202 {ticketId}, then GET /join/{ticketId} queued: skill or party matchPOST /sessions/{sessionId}/leaveinternal: /fleet/allocate, /fleet/servers/{id}/heartbeat, /sessions/{id}/reserve one regional site (x ~12)sessions are regional;cross-region playpays the RTT FleetManager warm processes, bin-packed by loadexpects heartbeats every 1 s,dead after 3 missedallocate, restore last full + deltas GameServers authoritative foranything scored or spent60 Hz simulate,20 Hz replicateup to 100 players each,8 processes per host SessionProcess CheckpointStore regional object storeengine + declared state,~0.5-1 MBdelta every 10 s,full every 60 s 4b. Matchmaker asksFleetManager to allocatea session warm process, registered,then reserved registers newsessions, marksdead ones 7a. GameServer heartbeatsFleetManager every 1 s dead after 3 missed (~3-5 s) 7b. GameServer checkpointsthe session every 10 sto CheckpointStore loss bound after a crash: <= 10 s PlayerDataStore durable progress, inventorywrite-behind: every 60 s+ on leaveper-server request budgets EconomyService purchases: idempotentreceiptoff the tick loop,retried until grantedthe session never holdsthe only copy TelemetryPipeline inputs + outcomes loggedoffline anti-cheat:outliers, replaymoderation seam MessageBus leaderboards, chat,teleportbetween sessionspub/sub, rate-limited 8a. GameServer writesprogress behind toPlayerDataStore every 60 s + on leave,events batched 8b. GameServer callsEconomyService forpurchases, off thetick loop idempotent receipt,retried until granted 9. GameServer streamsinputs and outcomesto TelemetryPipeline offline anti-cheat:outliers, replay 10. GameServers tradecross-session eventsover MessageBus never through thesession loop the not-so-happy paths: a GameServer crashes: FleetManager marks the session dead after 3 missed heartbeats, restores the last full plus deltas on a warm process, and the registry holds each seat for the 60 s token TTL; clients see the socket die, show 'reconnecting', and RESUME with the same handle and their resumeToken; loss <= 10 s of play and declared progress, never a purchasean EdgePoP dies: anycast rediscovers, the flow is re-pinned; the session survives because no game state was at the edgea hot launch: allocation is the hot part, not the registry key; pre-warm the fleet from the launch calendar, admit through a ticket queue with push, never a retry storman input fails validation (speed, cooldown, rate, ownership): clamped or dropped, never trusted; the client converges (smoothed) on the next DELTA the numbers this shape has to survive: 5M concurrent, ~167K live sessions (avg 30, max 100) ~21K hosts, ~12 regional sites, a few hundred EdgePoPs60 Hz simulate, 20 Hz replicate; INPUT 30 Hz<= 20 KB/s down per client, ~3 KB/s upup to ~0.8 Tbit/s at the per-client cap (5M x 160 kbit/s)p95 RTT <= 60 ms by placement (assumed); ~150 ms capown actions 0 ms perceived; others seen ~200 ms afterheartbeat 1 s, dead at 3 missed; checkpoint every 10 scrash loss <= 10 s of play and declared progress, never a purchase
1) Clients -> edge: nearest PoP by anycast, pinned tunnel -> regional game server; PoP terminates, no game state; ~3 KB/s up, <= 20 KB/s down2) Control plane: Gateway -> Matchmaker (latency map + policy -> region) -> Session Registry (Redis + Lua, atomic reservation), else Fleet Manager allocates warm server3) Game servers: one session per process, ~167K sessions on ~21K hosts; heartbeat 1 s; sim 60 Hz, validate every input; deltas 20 Hz, cap 20 KB/s, landing ~14; lag comp cap 200 ms4) Side stores: Checkpoint (delta 10 s, full 60 s, ~0.5-1 MB); DataStore (progress every 60 s + on leave); Economy (purchase = idempotent receipt, off tick)5) Async: Telemetry + anti-cheat; Message bus for cross-session; nothing on the tick path6) Failure: 3 missed heartbeats -> restore last full + deltas on warm process -> RESUME on 60 s token, same handle; loss <= 10 s play + declared progress, never a purchaseInvariant: "nothing scored or spent becomes truth without passing server-side validation"

What is Expected at Each Level

Mid: server authoritative, inputs in, deltas outmatchmaking -> regional serversizes nothing: no tick, no bytes, no loss boundSenior: prediction + reconciliation, interpolation; ~200 ms to see another player at 50 ms RTT8 MB/s naive -> 38 KB/s interest -> ~14 KB/s vs 20 cap, deltas vs ACKed baselinecrash: checkpoint + rejoin; loss <= 10 s play + declared progress, never a purchase"edge terminates, does not simulate"Staff: triangle: latency, fairness, moderationlag comp cap 200 ms = 100 + 100, a choice (Valve Source 1 s)physics-ownership exception + Hyperion; Luau behind a platform seamlaunch: 1,667 joins/s trivial; 33K sessions = ~4,100 hosts vs ~1,750 -> pre-warm, queuehot standby: a tier, doubled compute; ~21K hosts, egress ~0.8 Tbit/s cap, ~0.56 expected, KPI
DONE WHEN: 60/20; ~14 KB/s under 20 KB/s cap; predict self, interpolate others; lag comp cap 200 ms; checkpoint delta 10 s, full 60 s, RESUME; "nothing scored or spent becomes truth without passing server-side validation"