Design a real-time multiplayer system (Roblox, notes)

Functional Requirements

1. Join: experience -> session near me, right people (fill | queue by skill or party)
2. Play: inputs change a shared world, seen within ~200 ms, own actions instant
3. Trust: no client asserts a hit, a score, or (competitive) a position
4. Survive: crash loses <= 10 s of play + declared progress, never a purchase; back in ~10 s

Tier: per-experience flagcompetitive: server simulates movement, hot standbycasual: client-owned movement under bounds checks

Below the line (out of scope):

Game logic + scripting: inside the tick, creator's problemVoice chat, economy, marketplace: purchases only, in SurviveAsset delivery: ugc-assets; client has meshes + texturesModeration DECISION: ML problem; enforcement plumbing onlyAccount + auth: signed identity arrives with join

Non-Functional Requirements

Fleet: 5M concurrent, ~167K sessions, ~21K hosts:5,000,000 / 30 avg = ~167K sessions (max 100)one session = one process, nothing shared167,000 / 8 per host = ~21K hosts (8: design assumption, 12-16 cores)21,000 / 12 sites = ~1,750 hosts per regionone core per session, 16.7 ms tick; replication tick: 100 clients × ~30 interest decisionsfew hundred edge PoPs: terminate transport, no game state(Roblox: max players configurable, 700 announced; verify)

Tick: 60 Hz simulate, 20 Hz replicate:SIMULATE 60 Hz = 16.7 msREPLICATE 20 Hz = 50 ms = every third tick, delta vs last ACKed baseline(Roblox: ~20 Hz replication, 60 Hz Heartbeat, 240 Hz physics substep; verify the 240)INPUT 30 Hz: ~40 B + 2 × ~15 B + 28 B UDP/IP = ~100 B × 30 = ~3 KB/s upclient interpolates the gap

Bandwidth: ~14 KB/s landed vs 20 KB/s cap:Goal: 20 KB/s down per client (160 kbit/s), cellularUnit: ~40 B per entity: pos 6, rot 8, vel 6, anim+flags 4, id 2, + change mask + slack; 28 B UDP/IP header per PACKETNaive:100 × 40 B = 4 KB per client per update× 20 Hz = 80 KB/s per client× 100 clients = 8 MB/s (64 Mbit/s) per sessionO(N^2): 100 × 100 × 20 = 200,000 entity-updates/sInterest management:near: 30 × 40 B × 20 Hz = 24 KB/sfar:  70 × 40 B × 5 Hz  = 14 KB/s38 KB/s - over goal+ Deltas:38 × ~1/3 = ~13 KB/swhy: change masks, most fields unchanged in 50 ms+ 28 B × 20 Hz = 0.6 KB/s headers= ~14 KB/s - under cap, nicePer session: 100 × 160 kbit/s = 16 Mbit/sFleet:cap:    5M × 160 kbit/s = ~0.8 Tbit/slanded: 5M × 112 kbit/s = ~0.56 Tbit/s - KPI: replication bytes per client

Latency: p95 RTT <= 60 ms (assumption), hard cap ~150 ms, ~12 regional sites:OWN: 0 ms perceived, client predictionOTHERS: ~200 ms even at 50 ms RTTinput wait 0-33 ms+ one-way 25 ms+ next tick 0-17 ms+ next replication 0-50 ms+ one-way 25 ms+ interpolation buffer 2 × 50 ms = 100 ms= 150-250 ms, ~200100 ms buffer: a choice; 1 snapshot + extrapolate, jitter

Fairness: lag compensation cap 200 ms:server rewinds by shooter's one-way latency + interpolation delay200 ms RTT player = 100 + 100, at capbeyond cap: laggy player suffers (Valve Source default 1 s)

Matchmaking: two budgets:fill-join: p95 < 2 squeue-match (skill or party): seconds to ~1 min; this page: PLACEMENT only

Fault tolerance: loss bound:Detect: heartbeat to Fleet Manager every 1 s, 3 missed = dead, 3-5 sCheckpoint: delta 10 s, full 60 s, regional store, ~0.5-1 MB per full; engine state + declared staterestore: last full + <= 5 deltas; RESUME token TTL 60 s, back in ~10 sLoss bound: <= 10 s of play + DECLARED progress, never a purchase; undeclared: 60 s DataStore writeDurable: purchases: idempotent receipt, off tick, retried; progress: DataStore every 60 s + on leavewhy: DataStore per-server request budgets

Availability: join number + session KPI:region fails: joins elsewhere, sessions inside lost (checkpoint store regional)join: 99.9 percentsession: crash rate per session-hour, not nines

Triangle: latency, fairness, moderationprediction: spends fairness, buys latencyserver validation: spends latency, buys fairness + moderationplacement: buys latency, within a regionclient-owned physics: buys server CPU, costs fairness

Defining the Core Entities

Player: {playerId, connection, inputSeq, resumeToken}, max one session at a timeExperience: {experienceId, maxPlayers <= 100, regionPolicy, mode: fill | queue(skill, party)}Session: {sessionId, experienceId, region, serverAddr, capacity, occupancy, state, tick}one session = one process, nothing sharedoccupancy reserved atomically in the Session Registry (Redis + Lua)GameServer: {region, capacity, health, lastHeartbeat}, heartbeat every 1 s to Fleet ManagerInput: {playerId, seq, clientTick, actions[]}"inputs are intents, not outcomes"Delta / Snapshot: {serverTick, ackSeq, baselineTick, changes[]}; snapshot = delta against nothingCheckpoint: {sessionId, tick, blob: engine state + declared state}, delta 10 s, full 60 s, ~0.5-1 MBRegion / EdgePoP: ~12 regions run game servers; few hundred PoPs terminate transport, no game state

API or System Interface

// control plane, HTTPS
POST /join {experienceId, playerId, party?, prefs?}
     -> 200 {sessionId, serverAddr, edgeAddr, joinToken}   // placed now (fill path); serverAddr is a routing handle, e.g. "euw1/s91"
     -> 202 {ticketId}                                       // queued (skill or party), then
GET  /join/{ticketId}                                        // push or poll until placed
POST /sessions/{sessionId}/leave

// internal, control plane
POST /fleet/allocate {experienceId, region}                  // -> a session on a warm server
PUT  /fleet/servers/{id}/heartbeat {load, sessions[]}       // every 1 s
POST /sessions/{id}/reserve                                  // atomic slot reservation in the registry

// data plane, a UDP-based reliable/unreliable channel protocol
C -> S  INPUT    {seq, clientTick, actions[]}               // 30 Hz, unreliable, carries the last 3 inputs, ~3 KB/s up
S -> C  DELTA    {serverTick, ackSeq, baselineTick, changes[]}  // 20 Hz, unreliable
S -> C  SNAPSHOT {serverTick, entities[]}                   // on join, or when the ACKed baseline is > 1 s old
S -> C  EVENT    {chat | kill | score | purchaseConfirmed}  // reliable channel
C -> S  RESUME   {sessionId, resumeToken}                   // on reconnect

Two planes:control: HTTPS; join, leave, allocate, heartbeatdata: UDP-based, 30 Hz up / 20 Hz down; retransmit worse than lossnative: UDP + thin reliability layer (Roblox: RakNet-derived)web: WebRTC data channels or WebTransportfallback: TCP 443; head-of-line blocking -> hitch at 20 Hzreliable channel: chat, kill, score, purchaseConfirmed; never positionJoin returns an address: client connects directly to game server via edgejoinToken: game server checks itedgeAddr: anycast to nearest PoP, then pinnedwhy: anycast route flaps break UDPserverAddr: opaque routing handle, <region>/<sessionId>, e.g. euw1/s91edge resolves it over backbone; client never learns a game-server IPsame handle on RESUME after crashJoin 200 vs 202: 200 fill-join, placed now; 202 queue ticket, push or pollnever retry POST /join in a loopINPUT: last 3 inputs redundant~40 B + 2 × ~15 B + 28 B header = ~100 B × 30 Hz = ~3 KB/s upno position field, intents onlyDELTA: ackSeq: inputs applied, client replays the rest; baselineTick: snapshot it is relative tobaseline > 1 s old -> full SNAPSHOTRESUME: data-plane, not a second jointoken TTL 60 s > detect 3-5 s + allocate + restoreregistry holds SEAT for the TTL; after 60 s, fresh join

1) Join: matchmaking as placement

Flow: client -> gateway -> Matchmaker -> region -> policy -> reserve slot -> {serverAddr, edgeAddr, joinToken} -> client -> edge PoP -> game serverRegion: client latency probe at app start, ~a dozen pings, cached; Geo-IP fallbackwhy: region decides RTTPolicy: fill | queue(skill, party)Fill: Session Registry, Redis keyed (experienceId, region), sorted set by free slotsreservation: Lua script, check + decrement atomic; TTL ~30 sjoin p95 < 2 s = registry read + Lua call + at worst one allocationEmpty: Fleet Manager, warm process (assets loaded), bin-pack fullest host with roomwhy: cold boot blows p95 past 2 sno capacity: next-best RTT regionAfter join: client -> edge PoP -> game server direct; matchmaker off the pathQueue: 202 + ticketId; match forms -> region minimising p95 RTT -> allocate -> reserve

2) Play: the authoritative loop, inputs in, deltas out

Flow: inputs -> validate -> apply -> advance 16.7 ms -> every 3rd tick: delta vs ACKed baseline -> prioritise -> byte budget -> send -> client rewinds to ackSeq, replays unackedTick: 60 Hz, 16.7 ms; replication every 3rd tick = 20 HzValidate: max speed, cooldowns, input rate, ownership; fail -> reject or clampexperience's server script behind the seam (Roblox: server Luau)Simulation: one copy, this process; truth for anything scored or spentReplication: delta vs last ACKed baseline -> prioritise by interest -> cut to byte budgetunreliable channel; lost -> next delta covers itcap 20 KB/s, landed ~14 KB/sClient:prediction: own inputs applied same frame, un-ACKed inputs keptreconciliation: each DELTA -> rewind own entity to ackSeq -> replay unacked; disagree -> smooth, snap past thresholdinterpolation: others rendered between 2 newest snapshots, 100 ms behindHit registration: server rewinds targets to what shooter saw (one-way + 100 ms interp), cap 200 msMovement:server-simulated + prediction: fair, server CPU per player; competitive defaultclient-owned physics + bounds checks: cheaper (Roblox), speed/fly exploits; casual tierSecond player sees: own 0 ms; other ~200 ms at 50 ms RTTsample 0-33 + up 25 + tick 0-17 + replication 0-50 + down 25 + interp 100 = 150-250 ms

3) Latency: regions and edge

Flow: input sample wait -> one way up -> next tick -> next replication -> one way down -> interpolation buffernetwork: only the two one-way legsLever 1, placement: region minimising p95 RTT of the session's players~12 regional sites; target p95 RTT <= 60 ms (assumption)Lever 2, edge: few hundred PoPs terminate client transport; pinned unicast tunnel over private backbone to regional serverloss recovery on the short first hop: a few msanycast: discovery + handshake only, then pin one PoPwhy: route flap mid-session breaks a UDP flowDDoS: client knows only {edgeAddr, session handle}Lisbon -> eu-west: ~8 ms last mile + ~15 ms backbone = ~23 ms RTT, under 60 ms targetbackbone down: public internet path, same server, worse jitter"edge terminates, does not simulate"PoP holds no game state; lose a PoP -> re-pin, RESUMEEdge does not fix: Sao Paulo on eu-west ~110 ms RTT, ~5× Lisbon, under 150 ms capsession is regional, cross-region play pays the RTTGlobal low latency: no; regional low latency + choose the region well

4) Survive: crash, checkpoint, rejoin

Flow: heartbeat 1 s stops -> 3 missed, 3-5 s -> registry: session dead, seats held TTL 60 s -> warm replacement, same region -> restore last full + <= 5 deltas -> registry repointed, same handle -> client RESUME -> SNAPSHOT -> play on, ~10 sCheckpoint: regional store; delta every 10 s, full every 60 sfull: engine entity state + declared checkpointable; ~0.5-1 MB, 100-player worldLuau heap not serialisable -> declare-checkpointable API requiredingest ceiling: 167K × ~1 MB / 10 s = ~17 GB/s fleet, ~1.4 GB/s per regionfulls only: 1 MB / 60 s = ~2.8 GB/s fleet; a third at 30-player averagePlayer: socket dies -> "reconnecting", freeze, no lobby -> one registry read via gateway -> RESUME with token -> SNAPSHOTloss: <= 10 s of play + declared progress, never a purchasetoken TTL 60 s > detect + allocate + restoreDurable: purchase: idempotent receipt to economy service, off tick; progress: DataStore every 60 s + on leavewhy: DataStore per-server request budgetsStore itself dead: <= 60 s progress + the session lostConcede: not seamless; Roblox does not restore sessions (documented: DataStore + rejoin). Hot standby: a tier, double compute.

1) Client-server against peer-to-peer, and the sync model that follows

Answer: server owns anything scored or spent; clients send intentslatency: placement + prediction; smoothness: interpolation; fairness: lag compensation; bandwidth: deltas

Model: each client simulates itself, tells others where it isSaving: simulation server only; each client still uploads to N-1 peers; STUN/TURN relays costLie: any client, any message; no moderation seamNetwork: NAT traversal unreliable; N(N-1)/2 links, 100 players = 4,950Legit P2P: deterministic lockstep + rollback (fighting, RTS); Luau on a physics solver is not deterministicPrompt: P2P "hard to moderate, cheat-prone"

Model: one server owns the world; inputs in, full state out at fixed rate; client draws last receivedWins: fair + moderatable, one place decides every inputCosts:every action waits a round trip: 60 ms RTT -> jump >= 60 ms after press, ~100 with tick + replication waitsO(N^2): 100 × 100 × 20 Hz = 200,000 entity-updates/s, 8 MB/s per session

Server: only truth for anything scored or spent, every input validated; client hides the round tripPrediction, own entity: input applied same frame, unacked kept; each DELTA: rewind to ackSeq, replay; disagree -> smooth, snap past thresholdInterpolation, others: between 2 newest snapshots, 100 ms behind; ~200 ms to see another act at 50 ms RTTLag compensation, server: rewind targets to what shooter saw (one-way + interp delay), cap 200 msDeltas, wire: vs last ACKed baseline; 40 B per entity quantised; ~1/3 bytes -> 20 Hz affordable; bandwidth lever, not latencyException: Roblox hands physics ownership of own character to the client -> speed + fly exploits -> Hyperion (Byfron, 2023) client anti-cheat (verify)server sim + prediction for competitive; client ownership only non-competitive physics or casual tier, never scoredTriangle: prediction spends a little fairness to buy latency"the server never adopts a predicted state, it only lets the client draw one"

2) 100 players in one session: what the network looks like

Answer: naive 80 KB/s per client, 8 MB/s per session, O(N^2); interest 38 KB/s; + deltas ~14 KB/s vs 20 KB/s capper session 16 Mbit/s; fleet 5M × 160 kbit/s = ~0.8 Tbit/s at cap, ~half at landed

Model: every 50 ms, every client gets the whole worldMath:100 entities × 40 B = 4 KB per client per update× 20 Hz = 80 KB/s per client× 100 clients = 8 MB/s (64 Mbit/s) per sessionO(N^2): 100 × 100 × 20 = 200,000 entity-updates/s; 2× players -> 4×Cost: 80 KB/s = ~290 MB/hour of a phone's plan, ~6× egress of landed 14 KB/sserver: 64 Mbit/s per process, 512 Mbit/s per 8-process host

Model: changes since client's last ACKed snapshot, per-field change masksSaving: ~1/3 of full state: ~25-30 KB/s per client - overQuantisation: already in the 40 B: pos 6, rot 8, vel 6, anim + flags 4, id 2, + change mask + slack28 B UDP/IP header per PACKET, not per entityStill O(N^2): every client hears every entity; smaller constant, same shape

Interest management: ~30 near at full rate, far at 5 Hz, rest not at all (Roblox: StreamingEnabled)Math:interest alone: 24 + 14 = 38 KB/s - overdeltas alone: ~25-30 KB/s - overboth: 38 × ~1/3 = ~13 + 0.6 headers = ~14 KB/s vs 20 capPriority accumulator + byte budget per update: cap is a guarantee, not an averageACKed baselines: lost delta covered by the next; baseline > 1 s old -> full SNAPSHOTCost: CPU per client per update -> alert on tick > 16.7 ms

3) Cheating, and why prediction is the crack it lives in

Answer: server validates every input vs bounds, decides every hit, lag comp cap 200 ms"the server never adopts a predicted state; it only lets the client render one"

Client says: "I am here", "I hit him"; server writes it downspeed hacks, teleports, aimbots, infinite score: one modified client awayP2P failure mode inside client-server

Inputs vs bounds: max speed, cooldowns, rate limits, target ownership -> reject or clampsame tick as the input, no extra round tripNot covered: hits under latency; inhuman but inside every bound

Hits server-side, lag comp cap 200 ms:rewind target by one-way latency + interpolation delay, test, returncap: fairness choice (Valve Source default 1 s)200 ms RTT (100 + 100): at cap; 400 ms: cannot hit a target that moved 300 ms agoWho validates: the experience's SERVER script (untrusted Luau) behind the platform seamseam: server scripts + RemoteEvents, only path to server statewalls: Luau sandbox (script timeouts, memory limits); process isolation per instance (our design); per-connection rate limitsclient-side anti-cheat: Hyperion (Byfron, 2023), publicly documentedwhy: character physics client-ownedINPUT schema: {playerId, seq, clientTick, actions[]}, no position field to lie inEdge: 300 inputs/s -> throttled; no session token -> never reaches simulation; (DDoS shield)Physics ownership only for unscored objects: barrel yes; character, projectile, score nevercasual tier concedes character ownership: where exploits areOffline behavioural anti-cheat: logged inputs + outcomes -> outlier pipeline (accuracy, reaction time), replay; ban or shadow flag, not in the tickModeration seam: chat, actions, events pass the server -> logged, filtered, enforcedTension: prediction is the crack; client renders unconfirmed state, server never adopts itconverge: small corrections smoothed, past error threshold snappedscored outcomes: server, validated inputs, server time

4) The server crashes mid-game

Answer: heartbeat 1 s, dead at 3 missed; checkpoint delta 10 s + full 60 s; restore on warm process, RESUME; loss <= 10 s, never a purchasehot standby fed each tick: tick or two of loss, double compute, per-experience tier

Process dies, sockets close: 100 players -> matchmakingprogress in session memory: gone; purchase applied before recorded: paid for nothingdifferent session, different people: competitive round lost

Fleet Manager: dead in 3-5 s; restore last full + <= 5 deltas (<= 10 s old) on warm process, same region; clients RESUME with tokensLoss bound: <= 10 s of play + declared progress, never a purchase; back in ~10 spurchases: idempotent receipt, off tick; progress: DataStore every 60 s + on leavePlayer sees: "reconnecting" ~10 s, same world, same people, rewoundGap: visible interruption; ranked 10 s rewind: real complaint

Checkpoint: delta 10 s, full 60 s, ~0.5-1 MB, engine + declared state; restore = last full + <= 5 deltasLuau heap not serialisable; Roblox does not restore sessions; 17 GB/s ingest ceiling at 100-player worldsCompetitive tier, hot standby by STATE STREAMING:input-log replay needs deterministic sim; Luau + physics solver + os.time + HttpService: notshadow process fed each replication tick; failover = tick or two; clients RESUME to shadowCost: double compute for a rare event -> per-experience tier, not defaultfleet KPI: crash rate per session-hourClient timeline:t=0:      socket dies, "reconnecting", last state frozent=3-10 s: retry registry read via gateway, handle unchanged~t=10 s:  RESUME with token -> full SNAPSHOT -> play on, rewound <= 10 stoken TTL 60 s > detect (3-5 s) + allocate + restore; seat held for TTL; after 60 s fresh joinFleet timeline:t=0:      crasht=1-3 s:  heartbeats missedt=3-5 s:  dead; marked dead in registry; seats held for token TTLt=5-8 s:  replacement on warm host; last full + <= 5 deltas restoredt=8-10 s: registry repointed, live; clients RESUME, same handle; loss <= 10 sPurchases vs progress:purchase: idempotent receipt (ProcessReceipt-style), off tick, retried; no double-grantprogress: DataStore every 60 s + on leave, key events batched; checkpoint covers the gapwhy: DataStore per-server request budgetsCheckpoint store itself died: back to matchmaking; lost: <= 60 s progress + the session; purchases safeWhole-region failure: new joins elsewhere via latency map; sessions inside lost, store is regionalunless cross-region store replication or cross-region standby99.9 percent join target: join can move, a session cannot

5) Millions of players: what is sharded by what, and where it still gets hot

Answer: "the session is the shard"; registry by (experienceId, region); launch: capacity spike, not registry spike1M joins / 600 s = ~1,667/s trivial; 33K sessions / 8 per host = ~4,100 hosts vs ~1,750-host region -> pre-warm from launch calendar

Add hosts, one matchmaker + one registry:game servers fine; control plane: every join through one placement pathlaunch: no open sessions -> empty path; Fleet Manager cold-boots -> falls behind; clients retry POST /join -> retry storm

Session = unit of scale: one per process, nothing shared; 167K sessions on 21K hostsControl plane sharded the same way:Matchmaker by (experienceId, region)Session Registry: Redis partitioned by the same keyFleet Manager per region: bin-packing, pre-warmed processes, autoscale on occupancyNot enough for launch: capacity spike inside ONE region of ~1,750 hosts

Hot LAUNCH: 1M players, one experience, 10 minutesjoins: 1,000,000 / 600 s = ~1,667/s fleet-wide; ~140/s per region key; ~800/s if one region takes halfRedis sorted set + Lua reservation: trivial - not the bottlenecksessions: maxPlayers 100 -> 10K (~17 starts/s); maxPlayers 30 -> 33K (33,000 / 600 = ~55/s)capacity: 33K / 8 per host = ~4,100 hosts vs one region of ~1,750 -> more than double(~20 percent of the 21K fleet only if spread evenly)one core per session -> real hardware - THE hot partPre-warm from launch calendar: hosts provisioned, warm asset-loaded processes registered before first joinspin-up rate ahead of 17-55 sessions/s - secondary knobQueue when allocation lags: join returns 202 + ticket, push on allocation; never a retry stormSalt the hot key: sorted set + Lua spread across Redis partitions; cost: fill quality - an extra, not the fixCross-session features over a message bus: publish and move on; nothing off-session on the 16.7 ms tick pathRoblox: MessagingService (pub/sub, rate-limited), MemoryStoreService (ephemeral shared state), TeleportService + ReserveServer, DataStoreService (durable, per-server budgets)Still hot, and the bill:Fleet Manager for the launch region: one process per region, queue in frontfleet ~21K hosts (one core per session, 8 per 12-16 core box, design assumption)egress ~0.8 Tbit/s at cap, ~0.56 expected at landed 14 KB/s -> replication bytes per client is the KPI

Four more dives, briefly

Transport: UDP + thin reliability layer (Roblox: RakNet-derived); unreliable: INPUT, DELTA; reliable: EVENTweb: WebRTC data channels or WebTransport; fallback TCP 443 costs head-of-line blockingINPUT carries last 3 inputs: ~40 + 2 × ~15 + 28 B header = ~100 B, ~3 KB/s -> single drop loses nothingTick rate: 60 Hz sim = 16.7 ms; 20 Hz replicate (Roblox: ~20 Hz replication, 60 Hz Heartbeat, 240 Hz physics substep, verify the 240)inputs 30 Hz; 60 Hz: ~8 ms less input wait for ~6 KB/s up - competitive tierreplication 40 Hz: ~28 KB/s -> priority budget clips, "the cap is a cap"; sim 120 Hz halves CPU headroomnot deterministic across machines (Luau, physics solver, os.time) -> standby streams state, no lockstepClock sync: client clock ahead by one-way latency + margin; server tick + RTT/2 from handshake, updated from DELTA serverTickserver jitter buffer 1-2 ticks; early or late beyond it: dropped (late = speed-hack vector)0-33 ms input sample wait at 30 HzObservability: alert on "tick over budget", not "CPU high"per session: tick p50/p99 vs 16.7 ms; bytes per client vs 20 KB/s (~14 expected); inputs rejected/s, dropped lateper region: RTT p95 <= 60 ms; heartbeat misses; crash rate per session-hour; checkpoint write latency; join p95 vs 2 shost 40 percent CPU, one session at 25 ms: stutter for 100 people

P2P: any client can lie, no seam every message passes throughno moderation, no hit or score validationscored or spent -> server owns itAuthority costs latency, bought back:placement: p95 RTT <= 60 msprediction: own entity, 0 ms perceivedinterpolation: others, 100 ms bufferlag compensation: cap 200 msanother player's action still ~200 ms late at 50 ms RTT"Cheaper", "faster":P2P saves the simulation bill, not client bandwidth (N-1 uploads; STUN/TURN still cost)N(N-1)/2 = 4,950 connections for 100 + NAT traversal - not reliably lowerLegit P2P: deterministic lockstep + rollback (fighting games, RTS); Luau on a physics solver is not deterministicRoblox exception: physics ownership of own character -> client (documented)why: speed / fly exploits; client anti-cheat (Hyperion, Byfron 2023)mine: server sim + prediction for competitive tier; bounds validation alwaysclient ownership: unscored, or casual tier, conceded out loudTriangle: latency, fairness, moderationP2P pays fairness + moderation for latencyclient-server pays latency, buys most of it back

Who: server, always (Roblox: experience's server script)client sends {fire, seq, clientTick}, never "I hit him"As of when: when the shooter SAW the targetrewind others by one-way latency + 100 ms interpolation delay, test, back to presentLag compensation, cap 200 ms:this shooter: 100 + 100 = 200 ms, at cap, compensated in full300 ms RTT: extra not compensateda choice, not a constant (Valve Source 1 s)why cap: else laggy attacker shoots into the past, victim shot around a cornerVictim's client never decides: hit = EVENT on reliable channel; health is server statePrediction: muzzle flash instant, damage only when server says"the server never adopts a predicted state, it only lets the client render one"Bounds first: cooldown -> dropped; rate limit -> throttled; then hit test

Unit: ~40 B per entity per update, quantised, 20 Hzpos 6, rot 8, vel 6, anim+flags 4, id 2, + change mask + slack28 B UDP/IP header per PACKETNaive:100 × 40 B × 20 Hz = 80 KB/s per client× 100 clients = 8 MB/s (64 Mbit/s) per sessionO(N^2): 200,000 entity-updates/sInterest only (Roblox: StreamingEnabled):30 × 40 B × 20 Hz = 24 KB/s70 × 40 B × 5 Hz  = 14 KB/s38 KB/s - over+ Deltas:38 × ~1/3 = ~13 KB/swhy: change masks, most fields unchanged in 50 ms+ 28 B × 20 Hz = ~0.6 KB/s headers~14 KB/s vs 20 KB/s (160 kbit/s) cap - under, both levers neededPriority accumulator + byte budget per update: cap is a guarantee, not an averageACKed baselines: lost delta covered by the next; baseline > 1 s -> full SNAPSHOTPer session: 100 × 160 kbit/s = 16 Mbit/s worst caseFleet: 5M × 160 kbit/s = ~0.8 Tbit/s cap; ~0.56 Tbit/s at landed 14 - KPIUpstream: ~100 B × 30 Hz = ~3 KB/s per clientCost: CPU per client per update -> alert on tick > 16.7 ms; egress is the bill

Fleet:0-3 s:  heartbeats stop3-5 s:  3 missed -> dead; session marked dead in registry; seats held for token TTL5-10 s: new process on warm host; restore last full + <= 5 deltas; registry repointed, same handlePlayer:0:     "reconnecting", freeze on last state~10 s: RESUME with token (TTL 60 s) -> SNAPSHOT -> play onloss:  <= 10 s of play + declared progress, never a purchaseDurable: purchases = idempotent receipt, off tick; progress = DataStore every 60 s + on leaveStore itself dead: back to matchmaking, <= 60 s progress lostConcede: not seamless; Roblox does not restore sessions. Hot standby: a tier, double compute.

"the session is the shard"5M / 30 per session = ~167K sessionsone process, one core; 8 per host (design assumption, 12-16 core)~21K hosts, ~12 regionsnothing shared -> horizontal, no coordinationControl plane:Matchmaker: sharded by (experienceId, region)Session Registry: Redis, same partitionFleet Manager: per region; bin-packing, pre-warm, autoscale on occupancyLaunch: 1M players, one experience, 10 min = ~1,667 joins/s~140/s per region key spread; ~800/s if one region takes halfregistry NOT the bottleneckHot: regional CAPACITY:maxPlayers 100: 10K sessions (~17 starts/s)maxPlayers 30:  33K sessions (~55 starts/s)33K / 8 = ~4,100 hosts vs ~1,750-host launch region - more than doubleanswer: pre-warm from launch calendar; spin-up rate second; ticket queue with push, never clients retrying POST /joinCross-session: message bus, never the session loopMessagingService, MemoryStoreService, TeleportService + ReserveServer, DataStoreService (per-server request budgets)nothing outside the session on the 16.7 ms tick pathHot by design: Fleet Manager, launch region; one per region, queue in frontBill: ~21K hosts; egress ~0.8 Tbit/s cap, ~0.56 expected -> replication bytes per client is the KPI

Final Design

1) Clients -> edge: nearest PoP by anycast, pinned tunnel -> regional game server; PoP terminates, no game state; ~3 KB/s up, <= 20 KB/s down2) Control plane: Gateway -> Matchmaker (latency map + policy -> region) -> Session Registry (Redis + Lua, atomic reservation), else Fleet Manager allocates warm server3) Game servers: one session per process, ~167K sessions on ~21K hosts; heartbeat 1 s; sim 60 Hz, validate every input; deltas 20 Hz, cap 20 KB/s, landing ~14; lag comp cap 200 ms4) Side stores: Checkpoint (delta 10 s, full 60 s, ~0.5-1 MB); DataStore (progress every 60 s + on leave); Economy (purchase = idempotent receipt, off tick)5) Async: Telemetry + anti-cheat; Message bus for cross-session; nothing on the tick path6) Failure: 3 missed heartbeats -> restore last full + deltas on warm process -> RESUME on 60 s token, same handle; loss <= 10 s play + declared progress, never a purchaseInvariant: "nothing scored or spent becomes truth without passing server-side validation"

What is Expected at Each Level

Mid: server authoritative, inputs in, deltas outmatchmaking -> regional serversizes nothing: no tick, no bytes, no loss boundSenior: prediction + reconciliation, interpolation; ~200 ms to see another player at 50 ms RTT8 MB/s naive -> 38 KB/s interest -> ~14 KB/s vs 20 cap, deltas vs ACKed baselinecrash: checkpoint + rejoin; loss <= 10 s play + declared progress, never a purchase"edge terminates, does not simulate"Staff: triangle: latency, fairness, moderationlag comp cap 200 ms = 100 + 100, a choice (Valve Source 1 s)physics-ownership exception + Hyperion; Luau behind a platform seamlaunch: 1,667 joins/s trivial; 33K sessions = ~4,100 hosts vs ~1,750 -> pre-warm, queuehot standby: a tier, doubled compute; ~21K hosts, egress ~0.8 Tbit/s cap, ~0.56 expected, KPI