Design a real-time multiplayer system (Roblox)

Functional Requirements

1. Join: a player asks to play an experience and is placed into a session near them, with the right people (fill an existing session, or match by skill or party when the experience wants it).
2. Play: a player's inputs change a shared world that every player in the session sees within about two hundred milliseconds, and the player's own actions feel instant.
3. Trust: no client can assert a hit or a score, nor a position in the competitive tier; client-owned movement is a named concession under bounds checks and behavioural detection.
4. Survive: a game server crash mid-session loses at most ten seconds of play and of declared progress, never a purchase, and the players are back in within about ten seconds.

Tier is a per-experience flag the creator sets: competitive (server simulates movement, hot standby available) or casual (client may own its movement under bounds checks). The FRs and dives use those two words with exactly that meaning.

Below the line (out of scope):

• The game logic and scripting itself. I care that a session runs a simulation at a fixed tick and validates inputs; what the experience does inside that tick is the creator's problem, not this hour's.
• Voice chat, and the economy and marketplace. Purchases show up exactly once, in the survive requirement, because they change where durable state has to live, and then they go away.
• Asset delivery, which is its own topic: ugc-assets. This page assumes the client already has the meshes and textures it needs to draw the world.
• The moderation DECISION for chat and content is a machine-learning problem. Only the enforcement plumbing is mine: because everything passes through the server, actions and chat can be logged, filtered and enforced, and I will say where that seam is.
• Account and auth. I assume a signed player identity arrives with the join request.

Non-Functional Requirements

• Fleet: 5M concurrent, ~167K sessions, ~21K hosts. Size it first, because "how many sessions" is the number that shapes everything else. Call it 5M concurrent players at peak; Roblox's real figures are larger, but the shape is the same.
  • Sessions hold up to 100 players; assume 30 on average. So 5,000,000 / 30 is about 167K live sessions at peak.
  • One session is one server process, because nothing is shared between sessions and that is what makes them the unit of scale. At 8 processes per host that is about 21K hosts (167,000 / 8), spread over about a dozen regional sites, so roughly 1,750 hosts per region (21,000 / 12).
  • CPU sizing: one core per session, with a 16.7 ms tick budget and the replication ticks the heavy ones (100 clients x about 30 interest decisions and delta encodes each). So 8 sessions per host is a DESIGN ASSUMPTION for a 12-16 core box with headroom, not a Roblox fact.
  • A few hundred edge PoPs (points of presence) sit in front of those dozen sites. The PoPs terminate the client's transport; they hold no game state.
  • Label the Roblox specifics as publicly documented rather than known: max players per server is configurable and well above 100 (700 was announced; verify), and the 100 here is an interview ceiling for the bandwidth math.
• Then the tick: 60 Hz simulate, 20 Hz replicate. It sets the unit of everything downstream.
  • The server SIMULATES at 60 Hz, one tick every 16.7 ms. It REPLICATES at 20 Hz, sending each client a delta (only what changed since a baseline, the last snapshot that client acknowledged) every 50 ms, which is every third tick. Roblox is publicly documented at roughly that shape: replication about 20 Hz, a 60 Hz Heartbeat frame event (Roblox's name for its per-frame step, not the fleet health heartbeat), and a 240 Hz physics substep (verify the 240).
  • Clients send inputs at 30 Hz. The newest input is about 40 bytes, the two redundant older inputs ride along delta-encoded at about 15 B each, plus the 28 B UDP/IP header, so about 100 B on the wire x 30 Hz is about 3 KB/s upstream. Upstream is never the problem; downstream is.
  • Say why the two rates differ: simulation wants fine steps for physics and hit detection, replication is bounded by bandwidth, and the client interpolates across the gap.
• Bandwidth: 14 KB/s landed against a 20 KB/s cap, and do the arithmetic out loud rather than saying "efficient".
  • One replicated entity is about 40 bytes per update, and that 40 B is ALREADY quantised (rounded to fewer bits): position 6, rotation 8, velocity 6, animation and flags 4, entity id 2, plus a per-field change mask and some slack. The packet header (about 28 B UDP/IP) is per PACKET, not per entity.
  • Naive full-state broadcast to 100 players: 100 entities x 40 B is 4 KB per client per update, x 20 Hz is 80 KB/s per client, x 100 clients is 8 MB/s (64 Mbit/s) per session outbound. It is O(N^2): 100 x 100 x 20 is 200,000 entity-updates a second for one session.
  • Interest management (sending each client only the entities near it) alone, about 30 entities in full-rate interest and the rest at 5 Hz or not at all: 30 x 40 B x 20 Hz is 24 KB/s, plus 70 x 40 B x 5 Hz is 14 KB/s, so 38 KB/s raw. Not enough on its own.
  • Deltas against the last ACKed (acknowledged) baseline, on top of interest, cut the 38 to about a third, because the change masks show most fields do not change in 50 ms: about 13 KB/s of payload, plus about 0.6 KB/s of packet headers (28 B x 20 Hz). So it lands around 14 KB/s against a cap of 20 KB/s down per client (160 kbit/s), which a phone on cellular handles. Both levers are needed to get there.
  • Per session the 100-player WORST CASE is 100 x 160 kbit/s = 16 Mbit/s; a 30-player session has little to interest-manage and sits well under that. Fleet at peak, counted per player because that is the honest base: up to 5M x 160 kbit/s = 800 Gbit/s, about 0.8 Tbit/s at the per-client cap, and expected roughly half that, about 0.56 Tbit/s at the landed 14 KB/s; the KPI is the landed number. Egress dominates the bill, which is why replication bytes per client is the KPI.
• Latency: p95 client-to-server RTT at or under 60 ms, and be precise about which latency you mean, because own-action latency and other-player latency are different numbers.
  • The RTT budget: at or under 60 ms at p95 via regional placement across about a dozen sites, and label that an assumption (it is plausible for North America and Europe, not for every player); the hard cap for a playable session is about 150 ms.
  • The player's OWN actions feel instant, 0 ms perceived, because the client predicts them (applies its own inputs locally before the server confirms).
  • OTHER players' actions are seen roughly 200 ms after they happen even at 50 ms RTT, and that number is honest: input sample wait 0-33 ms (30 Hz) + one-way 25 ms + wait for the next tick 0-17 ms + wait for the next replication 0-50 ms + one-way 25 ms + a client interpolation buffer of two 20 Hz snapshots, 100 ms. That is 150-250 ms, call it 200. The 100 ms buffer is a choice: some engines hold one snapshot and extrapolate, and pay for it in jitter.
• Fairness: lag compensation capped at 200 ms, and the cap is a choice.
  • Lag compensation rewinds the server by the shooter's one-way latency plus their interpolation delay, capped at 200 ms. A 200 ms RTT player at 100 + 100 sits exactly at the cap, which is why 200. Beyond the cap the laggy player is the one who suffers, and that is a fairness choice (Valve's Source engine defaults to 1 s), not an industry constant.
• Matchmaking is two different products with two different budgets.
  • Fill-join (pick an open session near me): p95 under 2 s.
  • Queue-match (skill or party): seconds to about a minute. The queue mechanics are the game-matching topic; this page covers PLACEMENT and hands the ticket off.
• Fault tolerance has to come with a loss bound, or "resilient" means nothing.
  • Game servers heartbeat to the Fleet Manager (the per-region control service that allocates session processes and watches their heartbeats) every 1 s and are declared dead after 3 missed, so about 3-5 s.
  • The session checkpoints to a regional store, and the loss bound falls out of four facts:
    • Cadence: a delta every 10 s and a full every 60 s.
    • Contents: engine entity state plus whatever the experience declares checkpointable, about 0.5-1 MB per full for a 100-player world.
    • Restore: the last full plus up to five deltas, then players rejoin via a resume token (a token the join reply carries so a client can reattach to its session; TTL, time to live, 60 s) within about 10 s.
    • Loss bound: at most 10 s of play and DECLARED progress, never a purchase; undeclared progress is bounded by the 60 s DataStore write.
  • Durable player state (purchases, progress, inventory) is NEVER only in session memory. Purchases are an idempotent receipt to the economy service, off the tick loop, retried until granted; progress is written to the player DataStore (Roblox's durable per-player key-value store) every 60 s and on leave, with key events batched into those writes, because the DataStore has per-server request budgets and "write on every event" is not affordable.
• Availability, stated as a join number and a session KPI, not two nines figures.
  • A region failing sends new joins elsewhere; sessions inside it are lost outright, because the checkpoint store is regional, unless that store replicates cross-region. Target 99.9 percent for join.
  • The session KPI is the crash rate per session-hour, not a nines figure, because a dropped session is felt by 100 people at once and what you can actually measure and drive down is how often a process dies.
• Name the triangle before the design, so every later choice can be placed on it: latency, fairness and moderation. Prediction spends a little fairness to buy latency; the server-side validation seam spends latency to buy fairness and moderation; placement buys latency for free but only within a region; client-owned physics buys server CPU at the cost of fairness, and is named as a concession.

Defining the Core Entities

• Player is the identity behind a connection: playerId, and while in a session, a connection, an input sequence number and a resume token. A player is in at most one session at a time.
• Experience is a game: experienceId, maxPlayers (up to 100), a region policy, and a matchmaking mode (fill, or queue by skill or party). The mode is what decides which branch the join takes.
• Session is one running world, and it is the unit of everything: sessionId, experienceId, region, serverAddr, capacity, occupancy, state, and the current tick.
  • One session is one server process, and nothing is shared between sessions. Say that here, because it is why the fleet scales horizontally with no coordination in dive 5.
  • Occupancy is a counter that two matchmaker workers can race on, which is why the Session Registry (the Redis index of open sessions, HLD 1) reserves slots atomically rather than reading and writing.
• GameServer is host plus process: region, capacity, health, last heartbeat. The Fleet Manager owns this table and expects a heartbeat every 1 s; the heartbeat is what makes "dead" a fact rather than a guess.
• Input is what a client sends: playerId, seq, clientTick, actions[]. An input is an intent, never an outcome. "I pressed jump" is an input; "I am at (x, y, z)" is not, and refusing the second one is the whole trust requirement.
• Delta / Snapshot is what the server sends: serverTick, ackSeq, entity changes since the client's ACKed baseline. A snapshot is a delta against nothing, sent on join and whenever the baseline is too old.
  • ackSeq is load-bearing: it tells the client which of its inputs the server has applied, so the client knows which unacked inputs to replay after it rewinds.
• Checkpoint is sessionId, tick, and a blob: engine entity state plus whatever the experience declared checkpointable through an API, because developer Luau (Roblox's Lua dialect that experiences are scripted in) heap state is not generically serialisable. A delta every 10 s and a full every 60 s to a regional store, about 0.5-1 MB per full for a 100-player world; a restore is the last full plus up to five deltas. It is what bounds the loss on a crash.
• Region / EdgePoP: a region is one of about a dozen sites that run game servers; an edge PoP is one of a few hundred points that terminate the client's transport and carry the flow over a pinned tunnel to the region. The distinction matters because only one of them holds game state.

API or System Interface

// control plane, HTTPS
POST /join {experienceId, playerId, party?, prefs?}
     -> 200 {sessionId, serverAddr, edgeAddr, joinToken}   // placed now (fill path); serverAddr is a routing handle, e.g. "euw1/s91"
     -> 202 {ticketId}                                       // queued (skill or party), then
GET  /join/{ticketId}                                        // push or poll until placed
POST /sessions/{sessionId}/leave

// internal, control plane
POST /fleet/allocate {experienceId, region}                  // -> a session on a warm server
PUT  /fleet/servers/{id}/heartbeat {load, sessions[]}       // every 1 s
POST /sessions/{id}/reserve                                  // atomic slot reservation in the registry

// data plane, a UDP-based reliable/unreliable channel protocol
C -> S  INPUT    {seq, clientTick, actions[]}               // 30 Hz, unreliable, carries the last 3 inputs, ~3 KB/s up
S -> C  DELTA    {serverTick, ackSeq, baselineTick, changes[]}  // 20 Hz, unreliable
S -> C  SNAPSHOT {serverTick, entities[]}                   // on join, or when the ACKed baseline is > 1 s old
S -> C  EVENT    {chat | kill | score | purchaseConfirmed}  // reliable channel
C -> S  RESUME   {sessionId, resumeToken}                   // on reconnect

• Two planes with deliberately different transports, and that split is the shape of the architecture. The control plane is HTTPS: join, leave, allocate, heartbeat, all rare, all fine with a round trip. The data plane is a UDP-based channel protocol running at 30 Hz up and 20 Hz down, where a retransmit is worse than a loss.
  • Native: plain UDP with a thin reliability layer (Roblox's is publicly a RakNet-derived protocol). Web: WebRTC data channels or WebTransport. TCP on 443 is the fallback where UDP is blocked, and it costs head-of-line blocking: one lost packet stalls every packet behind it, which for a 20 Hz stream means a visible hitch.
  • The reliable channel exists for the things that must arrive exactly once and in order: chat, kill, score, purchase confirmed. Position never goes on it, because a stale position is worthless by the time it is retransmitted.
• Join returns an address, not a session; the client then connects DIRECTLY to the game server through the edge, and never touches the matchmaker again. The joinToken is what the game server checks so that only a placed player can attach; the edgeAddr is the PoP the client discovers by anycast (one address announced from every PoP, the network delivers to the nearest) and then pins for the life of the session, because anycast route flaps would break a UDP flow.
• serverAddr is an OPAQUE ROUTING HANDLE, not a game-server IP. Its format is <region>/<sessionId>, for example euw1/s91; the edge PoP resolves it to the game server over the backbone, and the client never learns a game-server address. That is what makes "the address never leaks" true rather than aspirational, and it is the same handle a RESUME presents after a crash: the client never gets a new address, the edge simply resolves the old handle to the replacement.
• Join has two shapes, 200 and 202, because it is two products. Fill-join places you now. A skill or party queue returns a ticket, and the ticket is handed to the game-matching machinery, which pushes or is polled. Do not make the client retry POST /join in a loop; that is the retry storm dive 5 is about.
• INPUT carries the last 3 inputs redundantly, so one lost packet loses nothing. The newest input is about 40 bytes, the two older ones are delta-encoded to about 15 B each, plus the 28 B UDP/IP header, so the packet is about 100 B on the wire and about 3 KB/s up at 30 Hz. That is cheap, and it means the unreliable channel is safe for the one message that drives the whole simulation. Inputs carry no position field to lie in: they are intents, and the server decides the outcome.
• DELTA carries ackSeq and baselineTick, and both are for the client's benefit. ackSeq says which inputs the server applied so the client can replay the rest; baselineTick says which snapshot the delta is relative to. If the client's ACKed baseline is more than 1 s old the server stops sending deltas and sends a full SNAPSHOT, so a lossy link recovers on its own.
• RESUME is a data-plane message, not a second join, because the whole point is that the player lands on the replacement server without going through matchmaking again. The resume token has a TTL of 60 s, chosen to exceed detection (3-5 s) plus allocate plus restore with margin, and the registry holds the player's SEAT for that TTL so a fill-join cannot take it during the failover. After 60 s the player is a fresh join.

1) Join: matchmaking as placement

• Matchmaking here is a placement problem first and a pairing problem second. The client hits the gateway (the public HTTPS front door), the gateway hands the request to the Matchmaker, and the Matchmaker's first job is to pick a REGION, because region is what decides the RTT and RTT is what decides whether the session is playable.
• Region comes from the client's own latency probe, not from a guess.
  • On app start the client pings a beacon in each regional site and sends the resulting latency map with the join. That is a dozen small pings, cached for the app session.
  • Geo-IP is the fallback when the probe is missing or stale, and it is a fallback for a reason: geography is a poor proxy for network distance.
• The Matchmaker then applies the experience's policy: fill or queue, party constraints, skill if the experience asks for it. Policy decides which of the two paths below the join takes.
• Fill path, which is the common one: pick an open session near me, and reserve one slot atomically.
  • The Session Registry is Redis keyed (experienceId, region), a sorted set of open sessions scored by free slots, so "an open session in this region for this experience" is one read.
  • The reservation is a Lua script (a Redis-side atomic script): check the free count and decrement it in one atomic step, with a TTL of about 30 s on the reservation so a client that never connects gives the slot back. Two matchmaker workers cannot both hand out the last seat.
  • The reply is serverAddr (the routing handle, euw1/s91 style), edgeAddr and a joinToken, returned to the client back through the gateway. p95 under 2 s is a registry read, a Lua call and one allocation at worst.
• Empty path: no open session in that region, so ask the Fleet Manager for one. The Fleet Manager keeps warm processes (pre-started, assets loaded) on every host, bin-packs new sessions by load (fills the fullest host that still has room, so the fleet can shrink), registers the new session in the registry, and the Matchmaker reserves into it. Cold-booting a process on demand is what makes join p95 blow past 2 s, so the fleet is warm on purpose. No capacity in the region at all: place in the next-best RTT region, and the player is told.
• Then the client connects DIRECTLY to the game server, through the edge PoP of HLD 3 (Latency: regions and edge), and never speaks to the matchmaker again. The matchmaker is on the join path only, so it can be slow, restarted or sharded without a running session noticing.
• Skill or party queue: the ticket goes into the queue machinery, out of scope here beyond the handoff. The join returns 202 with a ticketId, the queue mechanics live in game-matching, and when a match forms it comes back to this same placement step: pick the region that minimises the p95 RTT of the players in the match, allocate, reserve.

2) Play: the authoritative loop, inputs in, deltas out

• The game server runs one loop at a fixed 60 Hz, and every tick does the same four things. Read the inputs that arrived since the last tick, validate each one, apply them to the simulation, advance the world by 16.7 ms.
  • Validation is per input and it is bounds: maximum speed, cooldowns, rate of inputs, ownership of the thing being acted on. An input that fails is rejected or clamped, never trusted.
  • The simulation is the truth for anything scored or spent. There is exactly one copy of the world, on this process, and it is the only place a score, a hit or a purchase is decided.
• Who validates: the experience's server script behind the platform's seam (dive 3 has the four walls). On Roblox the game server runs the developer's Luau (Roblox's Lua dialect that experiences are scripted in), so the platform cannot know what a "hit" means; it provides the seam and the walls, and the experience's SERVER script does the validating.
• Every third tick, at 20 Hz, replication runs, per client, and it is a budgeted job rather than a broadcast.
  • For each client: compute the delta between the world now and that client's last ACKed baseline, prioritise the changes by interest (near, visible, relevant), cut the list to the client's byte budget, send.
  • The delta rides the unreliable channel. If it is lost, the next delta is computed against the same old baseline and covers the gap, so nothing is retransmitted.
  • The budget is what caps a client at 20 KB/s no matter how busy the world is; the typical landed figure is about 14 KB/s.
• On the client, the player's own entity is predicted, and everyone else's is interpolated, and those are two different mechanisms.
  • Prediction: the client applies its own inputs immediately, so a jump happens the frame the button is pressed. It keeps every input the server has not ACKed yet.
  • Reconciliation: on each DELTA the client rewinds its own entity to the server's state for ackSeq, then replays the unacked inputs on top. If the server agreed with the prediction, nothing visibly moves. If it disagreed, the client smooths the correction over a few frames and snaps only past an error threshold.
  • Interpolation: OTHER entities are rendered between the two most recent snapshots, which is 100 ms behind the server at 20 Hz. That buffer is what makes other players move smoothly instead of teleporting every 50 ms.
• Hit registration happens on the server with lag compensation, and it is the one place the server deliberately looks backwards. When a shot arrives, the server rewinds the other entities to where the shooter SAW them (the shooter's one-way latency plus their 100 ms interpolation delay), tests the hit against that, then returns to the present. The rewind is capped at 200 ms, so a laggy shooter cannot hit into the past forever.
• Movement is the one place there is a real choice, so make it out loud. Server-simulated movement with client prediction is fair and costs server CPU per player; client-owned physics under server bounds checks is cheaper and is what Roblox is documented to do, and it is where speed and fly exploits come from. Default: server simulation with prediction for anything competitive; client ownership named as the concession for the casual tier.
• Say the honest number for what a second player sees. Own actions: 0 ms perceived. Another player's action: roughly 200 ms even at 50 ms RTT, because it is input sample wait (0-33) plus one-way (25) plus tick wait (0-17) plus replication wait (0-50) plus one-way (25) plus the 100 ms interpolation buffer, 150-250 ms. That is what "within about two hundred milliseconds" costs once you have counted the buffer, and saying so is better than being corrected.

3) Latency: regions and edge

• Start by saying where the milliseconds actually go, because it makes the levers obvious. Input sample wait, client to server one way, wait for the next tick, wait for the next replication, server to client one way, then the interpolation buffer. Only the two one-way legs are network; the rest is design.
• Lever one, PLACEMENT, is the biggest lever by far. Put the session in the region that minimises the p95 RTT of the players in it. With about a dozen regional sites the target is p95 RTT at or under 60 ms (an assumption, plausible for North America and Europe), and that is a placement outcome, not a network trick.
• Lever two, EDGE, is a few hundred PoPs that TERMINATE the client's transport near them and carry the flow over a pinned unicast tunnel across the private backbone (the operator's own network between sites) to the regional server.
  • The PoP holds per-flow transport state and does loss recovery on the short first hop, so a lost packet is recovered in a few milliseconds instead of a full client-to-region round trip.
  • Fewer congested public-internet hops, because the client's packets leave the public internet at the nearest PoP.
  • Anycast is for discovery and the handshake only; then the client pins one PoP for the life of the session, because an anycast route flap mid-session would break a UDP flow.
  • A DDoS shield in front of game servers whose addresses never leak: the client only ever knows edgeAddr and an opaque session handle.
  • Worked example, the one the diagram draws: a player in Lisbon on an eu-west server has about 8 ms RTT on the lossy last mile to the PoP plus about 15 ms RTT on the private backbone, about 23 ms RTT, inside the p95 target of 60 ms.
  • If the backbone link degrades: fall back to the public internet path, same server, worse jitter. The session does not move.
• Say explicitly: the edge TERMINATES, it does not SIMULATE. There is no game state at the PoP. Game state lives on exactly one server, so losing a PoP loses a transport hop and not the session; the client re-pins to another PoP and RESUMEs. An edge PoP that ran simulation would be a second source of truth, and this page has exactly one.
• Now the thing edge does NOT fix, which is the trade the question is really asking about. Two players on opposite sides of the world in one session share one server, and physics does not care about anycast. The Lisbon player is at about 23 ms and a Sao Paulo player on the same eu-west server is at about 110 ms RTT, about 5x, under the 150 ms cap but not close; that player's PoP carries the flow over its own pinned tunnel to the same server, but the backbone cannot shorten the ocean. The honest answer is that a session is regional, and cross-region play pays the RTT. Placement decides who pays it.
• So the answer to "how do you get global low latency" is that you do not; you get regional low latency and you choose the region well. That is the difference between this design and a CDN, and it is worth saying out loud.

4) Survive: crash, checkpoint, rejoin

• The server heartbeats every 1 s to the Fleet Manager and checkpoints the session to a regional store: a delta every 10 s, a full every 60 s. Those intervals are the whole recovery story: the heartbeat bounds how fast a death is noticed, the checkpoint cadence bounds how much play is lost.
• Be honest about what is in the checkpoint and what it costs, because "compact world state" hides two problems.
  • A full is engine entity state plus whatever the experience declares checkpointable through an API, about 0.5-1 MB for a 100-player world; developer Luau heap state is NOT generically serialisable. A restore is the last full plus up to five deltas.
  • Concede that Roblox itself does not restore sessions: its documented answer is DataStore plus rejoin another instance. Checkpoint and restore is a step past that, and it needs that declare-checkpointable API to be worth anything.
  • Cost: the ingest ceiling is about 17 GB/s fleet-wide if every session were 100 players (167K sessions x about 1 MB / 10 s), about 1.4 GB/s per region, and that assumes every checkpoint were a full-sized 1 MB; with fulls at 1 MB / 60 s (about 2.8 GB/s fleet-wide) plus small deltas the actual is well below that, and a third again at the 30-player average.
• Death is declared after 3 missed heartbeats, so about 3-5 s, and then the Fleet Manager runs one sequence.
  • Mark the session dead in the registry, so no new joins land on it, and hold every player's SEAT for the resume token TTL (60 s) so a fill-join cannot take it during the failover.
  • Allocate a replacement process on a warm server in the same region.
  • Restore the world from the last full checkpoint plus up to five deltas, at most 10 s old.
  • Update the registry: the handle is unchanged (it is region plus sessionId), the mapping behind it now points at the replacement, and the session is marked live again.
• The clients see the socket die and show "reconnecting"; they do not go back to the lobby. One registry read (via the gateway) asking whether the session is live again, then, with the SAME handle, which the edge now resolves to the replacement, RESUME on the data plane with their resume token, land on the new server, get a full SNAPSHOT, and play continues. Loss at most 10 s of play and of declared progress, back in within about 10 s. The 60 s token TTL is chosen to exceed detection plus allocate plus restore with margin.
• Durable state was never at risk, and say why with the two paths. A purchase is an idempotent receipt to the economy service, off the tick loop, retried until it returns granted, and the session never holds the only copy, so a crash cannot lose one or double-grant one. Progress is written to the player DataStore every 60 s and on leave, with key events batched into those writes because the DataStore has per-server request budgets; between those writes the checkpoint carries declared progress, so the loss bound on progress is the same 10 s. If the checkpoint store itself is down, the honest loss is up to 60 s of progress (the DataStore cadence) and the session.
• Concede that this is graceful degradation, not a seamless failover. Ten seconds of "reconnecting" is a visible event. A hot standby fed by state streamed each replication tick takes over in a tick or two, that is dive 4's Great, and it costs double compute for a rare event, which is why it is a tier and not the default.

1) Client-server against peer-to-peer, and the sync model that follows

• The answer in two lines: P2P has no simulation bill and any client can lie, so the moment a game is competitive or moderated the server has to own anything scored or spent.
• Then the latency that authority costs is bought back with placement and prediction; interpolation buys smoothness and lag compensation buys fairness at the latency you have; deltas buy bandwidth, not latency.

2) 100 players in one session: what the network looks like

• The answer in two lines: naive full-state is 80 KB/s per client and 8 MB/s per session, O(N^2), and it eats a phone's plan and the egress bill.
• Interest management alone gets to 38 KB/s raw; deltas on top of interest cut the 38 to about 13 KB/s of payload, so it lands around 14 KB/s with headers against a 20 KB/s cap, which is 16 Mbit/s per 100-player session at the ceiling and up to about 0.8 Tbit/s across the fleet at the cap (5M x 160 kbit/s), expected roughly half at the landed figure.

3) Cheating, and why prediction is the crack it lives in

• The answer in two lines: the server validates every input against physical bounds and decides every hit itself, so a client can only ever ask, never assert; on Roblox "the server" is the experience's server script behind the platform's seam.
• Prediction is exactly where cheats live, so the rule is that the server never adopts a predicted state; it only lets the client render one.

4) The server crashes mid-game

• The answer in two lines: heartbeat every 1 s and dead after 3 missed, checkpoint delta every 10 s and full every 60 s to a regional store, restore on a warm process and clients RESUME, so the loss is at most 10 s of play and of declared progress, never a purchase.
• A hot standby fed by state streamed each replication tick takes it to a tick or two of loss, and doubles compute for a rare event, so it is a per-experience tier, not the default.

5) Millions of players: what is sharded by what, and where it still gets hot

• The answer in two lines: the session is the shard, so game servers scale with no coordination; a launch is a capacity spike, not a registry spike.
• The control plane is sharded by (experienceId, region), and the launch is REGIONAL: 1M joins in 10 minutes is about 1,667 joins a second fleet-wide, trivial for the registry, but 10K to 33K new sessions is about 4,100 hosts, more than a doubling of the ~1,750-host region a launch actually lands in, so pre-warm from the launch calendar, spin-up rate second, and a ticket queue with push.

Four more dives, briefly

• Transport: UDP with a reliability layer against WebRTC, WebTransport and TCP, and why head-of-line blocking is the deciding fact.
  • TCP delivers in order, so one lost packet stalls everything behind it until the retransmit arrives, which at 20 Hz is a visible hitch. Position data does not want that: a stale position is worthless by the time it is retransmitted.
  • Native clients: plain UDP with a thin reliability layer (Roblox's is publicly a RakNet-derived protocol), unreliable channel for INPUT and DELTA, reliable channel for EVENT. Web clients: WebRTC data channels or WebTransport, which give the same two channel kinds inside a browser.
  • TCP on 443 is the fallback for networks that block UDP, and it costs exactly the head-of-line blocking above; say so rather than pretending it is equivalent.
  • INPUT carries the last 3 inputs redundantly (newest about 40 B, the two older ones delta-encoded to about 15 B each, plus the 28 B header: about 100 B on the wire, about 3 KB/s), so the unreliable channel loses nothing on a single drop, and DELTAs are against ACKed baselines so a lost delta is covered by the next one.
• Tick rate: why 60 simulate and 20 replicate, and what a higher tick costs.
  • 60 Hz simulation is 16.7 ms per tick, fine enough for physics and hit tests to feel right; 20 Hz replication is a bandwidth decision, because every replication is a per-client delta computation and a packet. Roblox is publicly documented near that shape (about 20 Hz replication, a 60 Hz Heartbeat frame event, Roblox's name for its per-frame step and not the fleet health heartbeat, and a 240 Hz physics substep; verify the 240).
  • Why 30 Hz inputs when you simulate at 60: one input per two ticks. 60 Hz inputs would buy about 8 ms of mean input wait for double the upstream packets (about 6 KB/s); I would take 60 for a competitive tier.
  • Doubling replication to 40 Hz would want about 28 KB/s per client (about 14 KB/s landed, doubled), so the priority budget clips it and far entities degrade to lower rates: the cap is a cap. It also doubles the replication CPU per session for a smoother remote view that interpolation already mostly provides. Doubling simulation to 120 Hz halves the CPU headroom per tick, and tick time over budget is the alert that matters.
  • Determinism is worth one sentence here: the client's prediction runs the same movement code as the server so reconciliation rarely snaps, but a fully deterministic simulation across machines is not something this platform has (Luau, the physics solver, os.time), which is why the hot standby streams state rather than replaying inputs, and why lockstep P2P was never on the table.
• Clock sync and input timing: the client runs ahead so its inputs arrive just before the tick they target.
  • The client learns the server tick and RTT/2 from the handshake and keeps it updated from every DELTA's serverTick, then runs its own clock ahead by one-way latency plus a margin, so an input stamped for tick T arrives just before the server simulates T.
  • The server keeps a per-client jitter buffer of 1-2 ticks so a slightly early or late packet still lands in its tick. Early beyond the buffer is dropped; late beyond the buffer is dropped too, and that second rule is a security rule: accepting late inputs is a speed-hack vector, because a client that can post-date inputs can move twice in one tick.
  • This is also where the 0-33 ms input sample wait in the latency decomposition comes from: at 30 Hz the next input frame is up to 33 ms away, and it is honest to count it.
• Observability: the alert is "tick over budget", not "CPU high".
  • Per session: tick time (p50, p99 against the 16.7 ms budget), replication bytes per client per second against the 20 KB/s budget (about 14 KB/s expected), inputs rejected per second (a spike is either a cheat or a bug), inputs dropped as late.
  • Per region: RTT histograms per client (is p95 at or under 60 ms), heartbeat misses, crash rate per session-hour, checkpoint write latency, and join p95 against 2 s.
  • A host at 90 percent CPU with every tick under 16.7 ms is fine; a host at 40 percent CPU with one session ticking at 25 ms is a session that is visibly stuttering for 100 people. Alert on the thing the player feels.

Final Design

• 1) Player clients, native and web, discover the nearest Edge PoP by anycast and pin it (a few hundred PoPs, a terminating proxy with no game state), which carries the flow over a pinned tunnel across the private backbone to a regional game server. The client only ever knows the edge address and an opaque session handle; about 3 KB/s up, at most 20 KB/s down.
• 2) Control plane: the Gateway hands a join to the Matchmaker, which reads the client's latency map and the experience policy, picks the region, and either reserves a slot in an open session via the Session Registry (Redis, keyed by experience and region, Lua for the atomic reservation) or asks the regional Fleet Manager to allocate a session on a warm server. Queue-match tickets hand off to game-matching and come back to the same placement step.
• 3) The regional Game Server fleet runs one session per process, about 167K sessions on about 21K hosts at peak, and heartbeats the Fleet Manager every 1 s.
  • Simulate at 60 Hz; validate every input against bounds (the experience's server script, behind the platform's seam).
  • Replicate deltas at 20 Hz under a per-client cap of 20 KB/s, landing about 14; fleet egress at peak up to about 0.8 Tbit/s at the cap (5M x 160 kbit/s), roughly 0.56 expected.
  • Decide hits server-side with lag compensation capped at 200 ms.
• 4) Side stores, each with a different durability job.
  • The Checkpoint store (a regional object store): engine state plus declared state, a delta every 10 s and a full every 60 s, about 0.5-1 MB per full; it bounds the loss on a crash to 10 s of play and declared progress, never a purchase.
  • The Player DataStore holds durable progress: every 60 s and on leave, key events batched.
  • The Economy service takes purchases as an idempotent receipt off the tick loop, retried until granted, so no purchase ever exists only in session memory.
• 5) Async: every input and outcome is logged to the Telemetry and anti-cheat pipeline, and cross-session features (leaderboards, chat, teleport) ride a Message bus, so nothing outside the session is on the tick path.
• 6) Failure footnote: a crash is noticed after 3 missed heartbeats, the Fleet Manager restores the last full plus deltas on a warm process while the registry holds each seat for the 60 s token TTL, and clients see the session live again via the gateway and RESUME with the same handle and their tokens, the edge resolving it to the replacement; at most 10 s of play and declared progress lost, never a purchase, back in within about 10 s.
• The invariant worth closing on: nothing scored or spent becomes truth without passing server-side validation; latency is bought with placement and prediction; interpolation buys smoothness and lag compensation buys fairness at the latency you have; anything a client is allowed to own is bounded, named out loud as a concession, and either unscored or explicitly conceded for the casual tier. Every place the design spends latency, fairness or moderation, it says so.

What is Expected at Each Level

• Mid: says the server is authoritative and clients send inputs, and stops at "put servers near players".
  • Names the authoritative server, but not what authority costs in latency or how that is bought back.
  • Has matchmaking pick a regional server, and inputs in and deltas out.
  • Knows the server validates inputs, at least as a sentence.
  • Does not size anything: no tick, no bytes per client, no loss bound.
• Senior: prediction and reconciliation, the honest 200 ms, the bandwidth math, checkpoint plus rejoin with a loss bound, edge terminates not simulates.
  • Explains client-side prediction and reconciliation for the player's own entity, and interpolation for the others, and gives the honest roughly 200 ms number for seeing another player act at 50 ms RTT.
  • Does the bandwidth math (8 MB/s per session naive, 38 KB/s with interest management alone, about 14 KB/s landed against a 20 KB/s cap once deltas against the ACKed baseline are added) and reaches interest management on purpose.
  • Handles a crash with checkpoint plus rejoin and states the loss bound (at most 10 s of play and declared progress, never a purchase), with durable state kept out of session memory.
  • Says the edge terminates but does not simulate, and that game state lives on one server.
• Staff: the triangle and where each corner is paid, the 200 ms cap as a choice, the ownership concession with its cost, the regional launch arithmetic, standby as a tier, the bill.
  • Names the latency, fairness and moderation triangle and says where each is paid, rather than claiming to win all three.
  • Gives the 200 ms lag-compensation cap and the reason: a 200 ms RTT player at 100 + 100 sits at it, beyond it the laggy player suffers, by design, and it is a choice, not a constant.
  • Raises the physics-ownership exception unprompted, with its cost, and confines it to unscored objects or the explicitly conceded casual tier; knows the server that validates is running the developer's untrusted Luau behind a platform seam (RemoteEvents, script timeouts and memory limits), and that Roblox's client-side anti-cheat exists because of that concession.
  • Works the hot launch with the arithmetic: 1,667 joins a second fleet-wide is trivial for the registry; 10K to 33K sessions is about 4,100 hosts, more than a doubling of the ~1,750-host region a launch lands in, so the bottleneck is regional capacity: pre-warm on the calendar, spin-up rate second, admit through a queue.
  • Offers hot standby by state streaming as a per-experience tier, not the default, and charges for the doubled compute of that tier's sessions; knows input-log replay needs a determinism the platform does not have.
  • Puts the bill on the table: about 21K hosts (one core per session, 8 per host as a design assumption) and up to about 0.8 Tbit/s of egress at the per-client cap, roughly 0.56 expected; egress dominates, so replication bytes per client is the KPI.
  • Pushes back on the prompt where it is loose: edge servers do not lower the RTT between two players who chose to be far apart, "fallback to replication" is a doubling of compute you should charge for, and a checkpoint of a Luau world only holds what the experience declares.
  • Says "the server never adopts a predicted state, it only lets the client render one" as the sentence that reconciles prediction with fairness.