← all topics Messenger board · say it in this order 1 Problem2 Entities + API 3 High-level design4 Deep dives 5 Final + levelsreset

Design Facebook Messenger

Hello Interview flow, board rules · one idea per line · click a chip to mark where you are
SAY THIS SENTENCE FIRST

"Messenger is not a throughput problem, it is a routing and ordering problem. The hard parts are that the recipient is on a connection some other machine is holding, that the same person is on four devices which must all converge, and that a message may never be lost, duplicated, or reordered. So a per-conversation sequence number is the backbone, the client's own message id makes sends idempotent, and every device syncs by asking 'what have I not seen since seq N'."

STEP 1 OF 5

Understanding the problem

4 min

Functional Requirements

  1. Users should be able to send and receive messages in a 1:1 conversation, in near real time.
  2. Users should be able to read their history and have it match on every device they own.
  3. Users should be able to message a group.
  4. Users should be able to see delivery and read state.

Below the line (out of scope):

  • End-to-end encryption. Say it out loud as a deliberate cut, because it changes the server from a store that can read and index messages into one that cannot, and it makes multi-device sync a key-distribution problem rather than a storage problem.
  • Voice and video calls. Different transport, different problem (signalling plus media relay).
  • Search over message content, spam and safety classification, stories.

Non-Functional Requirements

Do the scale math first, because it shows which load is unusual
  • Concurrency sizes the unusual part of the system: about 750M live connections, held by about 15,000 nodes.
    • Take 2B monthly users, 500M of them online at once, and about 1.5 devices connected each: roughly 750M live connections.
    • A well-tuned connection node holds order of 100K sockets, so that is order of 7,500 nodes at perfect packing, and perfect packing is not a plan: with headroom and N+1 across availability zones call it 15,000 nodes that do nothing but hold sockets open.
    • That number is a cost of its own: holding sockets grows with the number of connected devices, not with the number of messages sent.
  • Messages are fewer than people expect: 100B a day, about 1.2M a second, with no globally hot key.
    • 100B messages a day is about 1.2M messages per second average, call it 3M at peak.
    • At roughly 200 bytes of text plus metadata that is order of 20 TB a day before replication, and it is append-only.
    • 1.2M writes/sec is a lot but it is evenly spread across billions of conversations, so there is no GLOBALLY hot key.
  1. Latency is the product: sender-to-recipient p99 under about 500ms while both are connected. Nobody forgives a chat app that feels slow.
  2. Durability is absolute, ordering is per-conversation.
    • Once the server has acked a send, that message must survive. Losing one is worse than being slow.
    • Ordering only has to be consistent within a conversation. There is no useful global order across conversations, and promising one would be a lie.
  3. Availability beats consistency for presence and typing indicators, which are disposable. It does not beat consistency for message delivery.
  4. Storage is trivial to reason about and expensive to hold: append-only, read overwhelmingly at the tail, and almost never deleted.
DONE WHEN: the interviewer has heard "the recipient is on a socket held by another machine, and the same human is on several devices". Those two sentences license the connection layer, the session registry, and the per-device sync cursor.
STEP 2 OF 5

Entities + API

4 min

Defining the Core Entities

  • User is the human: id, display name, and the devices they own.
  • Device is a single installation, and it is a first-class entity rather than a detail. Each device has its own sync cursor, because a phone that has been off for a week is at a different point in the story than the laptop that is open right now.
  • Conversation is the thread: id, type (direct or group), participant list, and last_seq. It is the unit of ordering and the unit of access control.
  • Message is one entry in a conversation: (conversation_id, seq) as the primary key, plus sender, body, created_at, and the sender's client_message_id.
    • The seq is the whole design in one column. It is assigned by the server, it is monotonic per conversation, and it is what ordering, gap detection, and resumable sync are all built on.
  • Participant joins a user to a conversation and carries that user's per-conversation state: their read_seq and delivered_seq cursors and their notification settings.
  • Receipt is delivery and read state. Model it as a cursor per participant, not as a row per message per recipient, or a 400-message backlog turns into 400 receipt rows per person. Note what this does not fix: one message still advances one cursor per participant.

API or System Interface

WS     /v1/connect                            // long-lived, authenticated, one per device

POST   /v1/conversations/{id}/messages        // { clientMessageId, body }
                                              // → 202 { messageId, seq, serverTime }
GET    /v1/conversations?since={cursor}       // the conversation list, changed ones first
GET    /v1/conversations/{id}/messages        // ?after={seq} for sync, ?before={seq} for scrollback
POST   /v1/conversations/{id}/read            // { seq } advances this user's read cursor
POST   /v1/conversations                      // { type, participantIds } → 201

// pushed down the socket, never polled for
event  message.new                            // { conversationId, seq, senderId, body }
event  receipt.updated                        // { conversationId, userId, deliveredSeq, readSeq }
event  presence.changed                       // { userId, state, lastActiveAt } - best effort
event  typing                                 // fire and forget, never stored
  • The send returns 202, not 201, and it returns the seq (the message's server-assigned position in its conversation).
    • 202 is honest: the server has durably accepted the message and taken responsibility for delivering it, which is not the same as having delivered it.
  • clientMessageId is supplied by the client and is the idempotency key. A phone that retries on a flaky network must not create two messages, and this is the only thing that prevents it.
  • Sync is one shape used everywhere: ?after={seq}. Scrollback is the same call in the other direction. A client that has been offline and a client that is scrolling up are running the same query.
DONE WHEN: you have said "the client supplies an id so retries are free, and the server supplies a seq so order and sync are free".
STEP 3 OF 5

High-level design

10 min, end to end, no dives yet

1) Users should be able to send and receive a message in real time

One send: the socket you are on is not the socket they are on SenderDevice sends over its socket to one ConnectionNode, which hands the message to MessageService. MessageService assigns the next seq and writes MessageLog, then acks 202 back to that node. It asks SessionRegistry who is connected and pushes to the different ConnectionNode holding the recipient's devices; a device with no live socket is woken through APNs or FCM and syncs. One send: the socket you are on is not the socket they are on the connection fleet holds sockets and computes nothing. a registry is what joins the two fleets. the connection fleet: ~7,500 nodes at full packing,call it ~15,000 with headroom ConnectionNode holds the WS ConnectionNode a different one SenderDevice one of theuser's devices TheirDevices phone + laptop MessageService stateless, scales on workdeploys without dropping750M sockets APNs / FCM third partypush notifications OfflineDevice syncs from its cursor MessageLog key (conversation_id, seq)append only, read at the tailevery message, in order SessionRegistry userId -> {(deviceId, nodeId)}TTL'd entries,refreshed by heartbeat 1. sends over its socket {clientMessageId, body} 2. hands it to the service 3. assigns the next seq, writes durable BEFORE the ack, not after 4. acks 202 {seq} 5. who is connected? 6. pushes to the node holding them per device, not per user 7. writes to the socket 8. no live socket? notify for each offline device 9. wakes it, it syncs
  • Split the fleet that holds connections from the fleet that does work.
    • Connection nodes terminate the WebSocket, authenticate the device, and do nothing else. They are memory-bound and almost stateless in what they compute.
    • The MessageService is a normal stateless service that is CPU and database bound.
    • Keeping them separate means you can deploy the logic without dropping 750M sockets, which alone justifies the split.
  • A SessionRegistry answers the only hard routing question: which node is holding this device's socket?
    • Key is userId, value is the set of (deviceId, nodeId) currently connected, with a TTL and a heartbeat.
    • Redis is a reasonable home: it is small and it is hot.
    • Keeping 750M entries alive by heartbeat is a large write load of its own; the dive on what melts first handles it.
  • The send path writes the message durably, then acks the sender, then pushes to recipients, in this order:
    • the sender's connection node hands the message to MessageService
    • MessageService assigns the next seq for that conversation and writes the message durably
    • only then does it ack the sender, because an ack before the write is a lie
    • it looks up every participant's devices in the SessionRegistry and pushes to the nodes holding them
    • devices that are not connected get a push notification instead, and will sync when they come back
  • A phone that retries after a lost ack would store the message twice; the dive on never losing or duplicating a message covers where the dedupe lives.
  • Write once, fan out to many. The body is stored one time in the conversation log, so a 50-person group stores one body and not fifty. The push itself does carry the body, because a client that had to fetch every message it was just told about would double the round trips.

2) History that matches on every device

Sync is one question: what is there after my seq? One conversation log holds seq 40 to 47. A phone whose cursor is at 41 asks for everything after 41 and gets 42 to 47; a laptop at 45 makes the identical call and gets 46 and 47. Each device keeps its own cursor. Sync is one question: what is there after my seq? a device gone for a week and a device gone for two seconds run the same query. there is no separate catch-up path. the conversation log, keyed (conversation_id, seq) 40 41 42 43 44 45 46 47 head = 47 the next send gets 48 PhoneDevice offline for a weekgets back 42..47 LaptopDevice open two seconds agogets back 46, 47 1. GET ?after=41 asks from its own cursor the phone is here: 41 2. GET ?after=45 the identical call the laptop is here: 45 one cursor per DEVICE. the same human is genuinely at two different points in the story. sync is two phases: a device holds one cursor PER CONVERSATION, and it has thousands of conversations.so the conversation list comes first: which conversations changed, and each one's last_seq.then the device pulls ?after={seq} only for the conversations it is behind on.
  • Every device tracks its own cursor, and sync is one question: what is there after my seq?
    • A device that has been offline for a week asks the same question as a device that was offline for two seconds.
    • There is no separate "catch up" path to get wrong, which is the point.
  • Sync is two phases, because the seq is per conversation and a device has thousands of conversations.
    • First the conversation list returns which conversations changed and what each one's last_seq now is, and only then does the client pull tails for the ones it is behind on.
    • A device with 2,000 conversations that ran one query per conversation would turn every reconnect into 2,000 reads.
    • The alternative worth naming, because it is what Messenger actually does: a per-USER inbox sequence layered over the per-conversation one, so a device really does hold a single integer and sync really is one call. It costs an inbox write per participant per message, which is the fan-out the group cap is paying for.
  • A push dropped on its way to a connected device leaves that device behind until it next syncs; the dive on what melts first covers how a client notices.
  • Store messages keyed by (conversation_id, seq) in a wide-column store.
    • The access pattern is a range scan over a partition key, which is exactly what Cassandra or HBase is for.
    • Reads are overwhelmingly at the tail, so the hot rows stay in cache and old history falls to cheap storage.
    • Facebook's own answer was HBase first and then MyRocks, and be careful quoting that: MyRocks is RocksDB underneath MySQL, so their history actually runs from a wide-column store TOWARDS sharded MySQL, not away from it.
      • What survives either choice is the access pattern and the engine shape: an ordered range scan over one partition key, served by a log-structured engine that likes append-heavy writes.
      • Saying that is stronger than name-dropping a store, and it does not fall over if the interviewer knows the actual history.
  • The conversation list is a separate, much smaller read. It is per-user, sorted by recency, and it is what the app opens to, so it should not require touching the message log.

3) Group conversations

A group is one write and N deliveries SenderDevice sends one message to MessageService, which assigns one seq and writes it once to MessageLog. MessageService then pushes it to every live device in the group: Ana, Bo and Cy here, while an offline device falls through to APNs or FCM. The group cap bounds the fan-out. A group is one write and N deliveries fan out on WRITE here, against the usual social-feed answer, because the product caps the group size. SenderDevice one message MessageService assigns ONE seqwrites ONCEthen fans out to devices MessageLog one log, one seq counterbodies stored once, not N times the fan-out, bounded by the product's group cap (250 in Messenger) Ana phone + laptop, both live Bo phone live Cy laptop live OfflineDevice falls through to APNs / FCM 1. sends 2. one write seq 512 for everyone 3. deliver to every DEVICE, not to every user 250 people at ~1.5 devices is ~375 pushes,most offline, so they become notifications where this stops working, and say it before they ask: a social feed fans out on READ because one author can have 100M followers. a group chat cannot.if the product ever wanted thousands, that is a broadcast channel, not a chat.the fix there is to change the PRODUCT: fan out on read, drop per-member receipts. not to scale the fan-out.membership changes are log entries too, so 'Ana added Bo' gets a seq and lands in the right place in the story.
  • A group is the same conversation object with more participants, which is what makes the seq model pay off. One log, one seq counter, one write, N deliveries.
  • Fan-out on write is right here because the product caps group size, the opposite of the usual feed answer.
    • A social feed fans out on read for celebrities because one author has 100M followers.
    • Group chats have a hard product cap, 250 in Messenger's case, so the fan-out is bounded by design rather than by hope.
    • Bounded fan-out means you can afford to do the work at write time and keep reads trivial.
  • Fan-out is to devices, not to users, so a 250-person group with 1.5 devices each is order of 375 pushes, and most of those are offline and become push notifications instead.
  • Membership changes are entries in the log too, so "Ana added Bo" gets a seq and arrives in the right place in the story rather than as a side channel.
  • A group of thousands is a broadcast channel, so change the product rather than scale the fan-out past the cap. A broadcast channel fans out on read and drops per-member receipts.

4) Delivery state, read state, and presence

Three states, three cursors, on one axis One conversation's seq axis runs from 40 to 48. read_seq sits at 42, delivered_seq at 45 and sent at 47; 48 is not sent yet. Two panels explain why a cursor replaces a row per message per recipient, and why presence stays off the message path. Three states, three cursors, on one axis sent is a fact about the server, delivered about a device, read about a human. two integers carry all three. the conversation's seq axis 40 41 42 43 44 45 46 47 48 48 not sent yet read_seq = 42 the human has seen up to here delivered_seq = 45 the MAX across their devices sent = 47 the server accepted up to here read ≤ delivered ≤ sent, always. the sender's UI colours every bubble at or below each cursor. Why cursors, not a row per message per recipient a row per message per recipient means a 400-messagebacklog is 400 rows each.two integers on the Participant row say the same thingin one write. but be precise: cursors collapse the per-MESSAGEdimension, not the per-PARTICIPANT one. one message still advances one cursor per participant:250 of them in a 250-person group. Presence is disposable, and off the message path a TTL'd key written by the ConnectionNode's heartbeat.alive means: there was a heartbeat in the last 30 seconds. do NOT fan presence out to every contact.publish only to people currently looking at aconversation with that person, or the writeamplification dwarfs the actual messages. if it is wrong for a minute, nobody is harmed.typing indicators are the same: fire and forget,never stored, never retried.
  • Sent, delivered and read are three different facts, about the server, a device, and a human:
    • sent is the server's 202 and it is a fact about the server
    • delivered is the recipient's device acking receipt and it is a fact about a device
    • read is the recipient's app being open at that message and it is a fact about a human
  • Carry them as cursors per participant, never as a row per message per recipient.
    • delivered_seq and read_seq on the Participant row say everything a per-message table would say, in two integers.
    • Participant is per user, but delivered is a fact about a device, so state the convention: delivered_seq is the MAXIMUM across that user's devices, which means the tick promises "it reached at least one of their devices", not all of them. Say that out loud, because it is a product promise and not just a storage detail.
    • Marking a 400-message backlog read is one write, and the sender's UI can colour every bubble at or below that seq.
  • Presence is best-effort and must never be on the message path.
    • It is a TTL'd key written by the connection node's heartbeat: alive means "there was a heartbeat in the last 30 seconds".
    • Do not fan presence out to everyone: publish only to people currently looking at a conversation with that person, or the write amplification dwarfs the actual messages.
    • If presence is wrong for a minute, nobody is harmed. Design it as disposable and say so.
  • Offline devices get a push notification through APNs or FCM, and the app syncs when it wakes. The dive on offline delivery covers what the notification carries.
DONE WHEN: the interviewer picks a box to open. Let them steer from here.
  • If they just nod: "the riskiest part is a message being lost or duplicated across retries and reconnects, shall I open that?"
  • Running behind: protect the dives on loss and duplication and on ordering. They contain the actual engineering.
STEP 4 OF 5

Potential deep dives

~20 min, interviewer steers
TRIGGER: "what if the network drops mid-send" / "can a message be lost or duplicated"

1) How is a message never lost and never duplicated?

CLIENT ID FOR DEDUPE · SERVER SEQ FOR ORDER · ACK AFTER THE WRITE
Bad Solution: fire and forget over the socket
  • The client writes to the socket and assumes it arrived.
    • A socket write succeeding means the bytes reached the kernel buffer, not that the server processed them, and certainly not that anything was stored.
  • What actually happens: the phone changes from wifi to cellular mid-send, the socket dies, and the message is gone with the UI still showing it as sent.
Good Solution: retry until acked
  • The client keeps the message in a local outbox and retries until the server acks. This does fix loss, which is why it is better than nothing.
    • But now the message can arrive twice: the server processed it and the ack was lost on the way back, so the client retries a message that already exists.
    • You have traded a lost-message bug for a duplicate-message bug, and duplicates in a chat are just as visible.
Great Solution: an idempotent send keyed by the client's own id
  • The client generates a clientMessageId (a UUID) before the first attempt and reuses it on every retry.
    • Where that uniqueness lives matters, and it is the detail people get wrong.
      • The message table is keyed by (conversation_id, seq), and a wide-column store has no unique secondary index, so you cannot just declare one on the client id.
      • Keep a separate dedupe record keyed by (conversation_id, sender_id, client_message_id) holding the seq that was assigned, written with a conditional insert (a lightweight transaction in Cassandra, or a row in whatever store already serialises the conversation).
      • Claiming a plain unique index on the log itself is the answer that falls apart the moment the interviewer asks which store enforces it.
    • On conflict the server does not error, it returns the seq it assigned the first time, so the retry looks like success to the client and reconciles to the same bubble.
  • Ack only after the durable write, never before. The 202 is a promise, so it must not be made until the message would survive the process dying immediately after.
  • Delivery to the recipient is at-least-once, and the seq makes that safe.
    • Pushing the same seq twice is harmless because the recipient already has it and drops it.
    • That is the trick: you do not need exactly-once delivery if the payload is idempotent to apply, and a keyed message is.
  • The client's local outbox has to be durable too, or a crash before the first send loses a message the sender saw as sent. Without it, that message never reaches any recipient, and the server has no record of it. A durable client outbox is a real piece of engineering people skip.
TRIGGER: "what does in order actually mean here" / "two people send at the same instant"

2) Ordering: what is it, and who assigns it?

SERVER ASSIGNS PER-CONVERSATION SEQ · CLIENT CLOCKS ARE NOT ORDER
Bad Solution: order by the client's timestamp
  • Client timestamps cannot order a conversation, because device clocks cannot be trusted.
    • Phone clocks are wrong, sometimes by minutes, and users can set them by hand.
    • Two devices in different timezones with skewed clocks produce a conversation that reads as a shuffled deck.
Good Solution: the server assigns a seq per conversation
  • The server assigns a monotonically increasing seq per conversation, and that is the order. This fixes clock skew, because no client clock is involved.
  • Still wrong if the seq is allocated in one step and the message written in another:
    • A process dying in between burns a number that nothing will ever fill.
    • A client holding 40 that sees 42 then asks for 41 forever.
Great Solution: a gapless seq, allocated on the conversation's shard in the same write as the message
  • The server assigns a monotonically increasing seq per conversation, and that is the order. Full stop.
    • Ordering is only promised inside a conversation, which is the only place it means anything.
    • Two messages sent at the same instant get two different seqs, and which one is lower is arbitrary but consistent for everyone.
  • Allocate the seq on the conversation's own shard; that counter is the one place a conversation is a hot key.
    • All writes for one conversation go to one shard, keyed by conversation_id, and the counter is incremented there.
    • Allocate it with a single writer per conversation or a conditional write that returns the value it set, not with a distributed lock.
      • Do not reach for a Cassandra counter column here: counter increments are not idempotent, so the retry this design guarantees would double-count, and you cannot read-and-increment atomically to learn the value you were given.
    • This is fine precisely because conversations are small: a busy group is a few messages a second, not thousands, three orders of magnitude below what one shard leader serializes.
  • The client shows an optimistic bubble before it has a seq, and must reconcile.
    • Render immediately with a local placeholder so the app feels instant, keyed by clientMessageId.
    • When the 202 comes back with a seq, slot the message into its true position, which may move it.
    • Getting this wrong is the classic bug where your own message jumps around after sending.
  • Gaps are detectable and therefore recoverable: a client that holds seq 40 and receives 42 knows exactly what to ask for. A design without seq numbers cannot even tell that it missed something.
  • The seq space has to be gapless, because a client recovering a gap asks for a number that must exist.
    • If the seq is allocated in one step and the message written in another, a process dying in between burns a number that nothing will ever fill.
    • A client holding 40 that sees 42 then asks for 41 forever, and the recovery path for the WHOLE design stalls on a message that does not exist.
    • So allocate inside the same write that stores the message, and if a hole can still happen, publish a per-conversation head watermark so a client can be told "nothing below 512 is coming" and move on.
TRIGGER: "I have four devices" / "why is my phone showing unread when I read it on desktop"

3) Multi-device sync, and the read cursor

CURSOR PER DEVICE FOR SYNC · CURSOR PER USER FOR READ
Bad Solution: one cursor that means both "synced" and "read"
  • Each device keeps one cursor and uses it for both how far it has caught up and what the human has seen.
  • Reading on desktop then cannot clear the badge on the phone, because the phone's cursor has not moved.
Good Solution: a sync cursor per device and a read cursor per user
  • Each device tracks how far it has synced, and the user has one read cursor, so reading on the laptop clears the phone's badge.
  • Still wrong:
    • A plain update lets read state move backwards: a phone that was offline delivers a stale "I read up to 30" after the desktop said 45, and the badge comes back from the dead.
    • A new laptop pulling four years of history eagerly is how you take down your own storage tier.
Great Solution: two cursors, a guarded advance, and a bounded backfill
  • The sync cursor (per device) and the read cursor (per user) are different, and conflating them is the bug.
    • Sync cursor is per device: how far this installation has caught up. The laptop and the phone are genuinely at different points.
    • Read cursor is per user: whether the human has seen it. Reading on desktop must clear the badge on the phone.
  • Adding a new device is a bounded backfill, not a full history download.
    • Sync the conversation list first, then the last screenful of each, then older pages lazily as the user scrolls.
    • A new laptop pulling four years of history eagerly is how you take down your own storage tier.
  • Read state must never move backwards, so the advance is a guarded conditional write (read_seq < :n).
    • The advance is a conditional write, and it must be scoped or it moves every row in the table:
      • UPDATE participant SET read_seq = :n WHERE conversation_id = :c AND user_id = :u AND read_seq < :n
      • delivered_seq needs the identical guard, since delivery acks arrive from several devices out of order and regress the same way.
    • Without the guard, a phone that was offline delivers a stale "I read up to 30" after the desktop said 45, and the badge comes back from the dead.
    • This is a compare-and-set: the same guarded-write shape as the seq allocation earlier, and it is worth naming as such.
  • A device gone past some threshold resyncs the conversation list and pulls tails, rather than replaying the gap message by message. Long-absent devices are a special case worth pre-empting.
TRIGGER: "what breaks first at this scale" / "a connection node dies"

4) What melts first, and what happens when a node dies

THE CONNECTION FLEET IS THE FRAGILE PART, NOT THE DATABASE
  • The database is the easy part: 1.2M writes/sec spread over billions of conversations has no hot key.
    • It is a sharding exercise, the easy kind of scale, and saying so early buys credibility.
    • Contrast with a ticketing system, where the whole problem is one row. Here, the whole problem is one socket.
Bad Solution: clients reconnect at once and nodes queue without limit
  • A connection node holding 100K sockets dies, and 100K devices reconnect at once, then each one runs a sync query.
  • Lose a whole availability zone and that is millions of simultaneous reconnects plus millions of catch-up reads.
  • A node with unbounded per-connection queues dies the moment a slice of clients stop draining, which on mobile is normal rather than exceptional.
Good Solution: jittered backoff, load shedding, and bounded sync pages
  • Jittered exponential backoff on the client, connection nodes that shed load rather than accept everything, and sync that returns a bounded page instead of everything since forever. This survives the reconnect storm.
  • Still wrong:
    • A client only notices a missed message when a LATER push arrives, so pushes that stop while the socket stays up leave it stale forever.
    • Every connect and disconnect is a registry write, so a flapping mobile network becomes registry load that has nothing to do with messages.
Great Solution: push is only a hint, the seq range is the truth, and the fleet is built to drop and recover
  • What actually melts is the reconnect storm.
    • A connection node holding 100K sockets dies, and 100K devices reconnect at once, then each one runs a sync query.
    • Lose a whole availability zone and that is millions of simultaneous reconnects plus millions of catch-up reads.
    • The fix is unglamorous and expected: jittered exponential backoff on the client, connection nodes that shed load rather than accept everything, and sync that returns a bounded page instead of everything since forever.
  • When a node dies, its devices reconnect to other nodes, re-register in the SessionRegistry, and sync from their cursors. The socket closes and the client reconnects through the load balancer. Nothing is lost because nothing durable lived on that node.
  • Real-time delivery is an optimization, not the source of truth.
    • The socket push is a hint that something arrived. The seq range is the truth.
    • If a push is dropped mid-stream, the client notices a gap in seq and pulls the range it missed.
    • But gap detection only fires when a LATER message arrives, so it cannot see tail loss, and tail loss is the common case: someone sends a last message and stops typing. Nothing later ever comes, so no gap is ever observed.
    • The conversation list is what actually covers that: it returns each conversation's current last_seq, so a client compares against its own position and needs no later message to notice. Run it on reconnect and on foreground.
    • This is why the design does not need reliable delivery over the socket, and that is a large simplification.
  • Session registry churn also melts, because every connect and disconnect is a registry write.
    • Every connect and disconnect is a write, so a flapping mobile network turns into registry write load that has nothing to do with messages being sent.
    • Short TTLs with heartbeats mean a dead node's entries expire on their own rather than needing cleanup, which is the same expiry-and-reclaim pattern as a lease.
    • Do the heartbeat arithmetic before calling the registry the cheap part, because it is not.
      • 750M devices on a 30-second TTL is order of 25M writes/sec, which is roughly 20x the entire message write rate.
      • So the node batches: one heartbeat per node covering all ~100K of its sockets, which turns 25M/sec into 7,500 writes per interval and stores liveness per node with a membership set under it.
      • The heartbeat rewrites the entry rather than just extending a TTL, so losing the registry heals in one interval instead of waiting for every client to reconnect, and costs one heartbeat interval of routing rather than a message.
  • The worse failure is silent: pushes stop while the socket stays up, so the client sees no gap and stays stale.
    • Recovery is "notice a gap in seq and pull", but a client only notices a gap when a LATER push arrives.
    • So if pushes stop while the socket stays healthy (a stale registry entry, a wedged outbound path, an evicted key), the client sees no gap, has no reason to reconnect, and sits quietly stale forever.
    • Push being only a hint is what makes dropped pushes survivable, but it also means the pull path needs a trigger that does not depend on a push: an idle resync, or a periodic server-sent head watermark per open conversation.
  • Bounded per-connection queues give backpressure almost for free: a node under pressure drops queued pushes and sends one resync marker.
    • A node holding 100K sockets with unbounded per-connection queues dies the moment a slice of clients stop draining, which on mobile is normal rather than exceptional.
    • So the outbound queue per connection is bounded, and here is the payoff: because a push is only a hint, a node under pressure can DROP the queued frames and send one resync marker instead.
    • Correctness is untouched, because the seq range is the truth. A design where the socket was the source of truth could not do that.
  • Pushes in flight when the node dies are lost: a push aimed at a dead node is simply dropped, and the client's post-reconnect sync is what recovers it. That is why the design tolerates unreliable pushes by construction.
TRIGGER: "how do notifications work when the app is closed"

5) Offline delivery and notifications

THE SOCKET IS THE FAST PATH, PUSH IS THE FALLBACK
Bad Solution: build the timeline from notification payloads
  • The app inserts each notification's message into the conversation as it arrives.
  • Push delivery is neither ordered nor guaranteed, so this is how you end up with messages in the wrong order, or missing.
Good Solution: the notification is a nudge, and the app syncs from its cursor
  • The notification carries the conversation id and seq, and the app syncs from its cursor rather than trusting the payload. This fixes order and loss.
  • Still wrong:
    • Twenty messages in a busy group become twenty banners.
    • A message the user already read on another device still shows its banner on this one.
Great Solution: a nudge that syncs, collapsed per conversation and withdrawn on read
  • The SessionRegistry lookup returns no live connection for a recipient's device, which is the whole trigger. Not connected means notify through the platform instead.
  • The notification carries the conversation id and seq, and the app syncs from its cursor rather than trusting the payload.
    • The payload carries conversation id and seq, and enough text for the notification to be useful.
    • The app treats the notification as a nudge and syncs from its cursor, because push delivery is neither ordered nor guaranteed.
    • Building the timeline from notification payloads is how you end up with messages in the wrong order.
  • Collapse notifications per conversation, so twenty messages in a busy group become one banner with a count. Collapse aggressively.
  • Respect the read cursor: if the user reads on another device, the notification on this one should be withdrawn, which is a real API on both platforms and a detail most candidates miss.
  • APNs and FCM are third parties with their own failure modes, so treat them as best-effort and never as the durable path. The durable path is always the message log plus the sync cursor.
DONE WHEN: you have opened at most two dives properly rather than five thinly, and the interviewer has heard why push being unreliable is a design choice rather than a weakness.
FLASHCARDS · THE FIVE HARDEST PROBESshow all (interview mode)
The recipient is connected to a different machine than the sender. How does the message get there?
  • MessageService looks the recipient up in the SessionRegistry, which maps userId to the set of (deviceId, nodeId) currently connected.
  • It then pushes to those specific nodes, which write to the sockets they hold.
  • The registry is Redis with TTL'd entries refreshed by each node's heartbeat, so a dead node's entries expire without a cleanup job.
  • If the lookup returns no live device, the message is already durably stored, so the fallback is a push notification and the device syncs on wake.
The client retries a send because the ack was lost. What stops a duplicate?
  • The clientMessageId, generated once before the first attempt and reused on every retry.
  • A dedupe record keyed by (conversation_id, sender_id, client_message_id), written with a conditional insert, holding the seq that was assigned.
  • Not a unique index on the message log itself: the log is keyed by (conversation_id, seq) and a wide-column store has no unique secondary index to declare.
  • On conflict the server returns the seq it assigned the first time rather than an error, so the retry is indistinguishable from success.
  • Delivery to recipients stays at-least-once, which is safe because applying the same seq twice is a no-op.
Two people send at the same instant. Who decides the order, and what does the client show?
  • The server does, by assigning a monotonic seq per conversation on a single shard keyed by conversation_id.
  • Client timestamps are never the order, because phone clocks are wrong and user-settable.
  • Which of the two gets the lower seq is arbitrary, but it is the same for every participant, which is what matters.
  • The sender rendered an optimistic bubble already, so it reconciles by clientMessageId when the seq arrives, and the bubble may move.
I read a message on my laptop. Why does my phone's badge clear, and how do you stop it coming back?
  • Read state is a cursor per USER, not per device, so advancing it on the laptop is what clears the phone.
  • Sync state is a cursor per DEVICE, which is a different thing: the phone genuinely has not caught up.
  • The advance is guarded AND scoped: UPDATE participant SET read_seq = :n WHERE conversation_id = :c AND user_id = :u AND read_seq < :n.
  • Without that guard a stale advance from a reconnecting device moves the cursor backwards and the badge returns.
A connection node holding 100K sockets dies. Walk me through the next thirty seconds.
  • Nothing durable was on it, so no message is lost: the log and the cursors are the truth.
  • 100K clients notice the socket close and reconnect through the load balancer to other nodes, with jittered backoff so they do not arrive as one wave.
  • Each re-registers in the SessionRegistry and syncs from its cursor, so the reconnect storm is really a read storm against the message store.
  • Pushes aimed at the dead node during the window are dropped, and the post-reconnect sync is what recovers them, which is why unreliable push is tolerable by design.
  • The dead node's registry entries expire on their own via TTL rather than needing a cleanup pass.
STEP 5 OF 5

Final design + what is expected at each level

wrap

Final Design

Messenger, end to end Devices send over a socket held by a ConnectionNode, which hands the message to MessageService. MessageService writes MessageLog before acking, advances cursors in ParticipantStore, asks SessionRegistry who is connected, and pushes to their nodes or notifies APNs / FCM. Devices also sync continuously with GET ?after={seq}; the node heartbeat writes PresenceStore. Messenger, end to end the seq is the backbone: it is the order, the gap detector, and the sync cursor, all in one column. Devices one socket eachown sync cursor the connection fleet, ~15,000 nodes ConnectionNode terminates the WS,authenticates, routesholds ~100K sockets,computes nothing MessageService assigns theper-conversation seqwrites, THEN acks 202fans out to devices APNs / FCM for devices withno live socketbest effort, carriesconversation + seq PresenceStore TTL'd key per user,heartbeat writtenpublished narrowly,never broadcastdisposable by design MessageLog wide-column,key (conversation_id, seq)append only, range scan for syncTHE TRUTH and the order ParticipantStore read_seq and delivered_seqper memberguarded advance, never backwardsone write marks 400 messages read SessionRegistry userId -> {(deviceId, nodeId)}one heartbeat per node,self-cleaninga lookup, never a message store 0. and continuously: GET ?after={seq} THIS is the truth. everything below it is the optimization. 1. sends {clientMessageId,body} 2. hands off 3. assigns seq, writes durable BEFORE the ack 4. advances the cursors delivered / read 5. who is connected? 6. push to their nodes per device, not per user 7. no socket? notify the heartbeat writes presence the not-so-happy paths: a dropped push costs nothing: a gap in seq, or a newer last_seq in the conversation list, sends the device for the range after its cursor.a lost ack costs nothing: the retry carries the same clientMessageId and gets back the same seq.a ConnectionNode dying costs a reconnect: nothing durable lived there, and its registry entries expire on their own.an AZ dying costs a reconnect STORM, which is the real failure here: jittered backoff, load shedding, bounded sync pages.presence being wrong costs nothing, which is why it is deliberately kept off this path.
  • Devices hold a long-lived WebSocket to a ConnectionNode, and the connection fleet is sized and deployed independently of everything else because it scales on sockets rather than on work.
  • The SessionRegistry maps userId to the set of connected (deviceId, nodeId), in TTL'd entries that one heartbeat per connection node rewrites, so failure cleans itself up.
  • MessageService assigns the per-conversation seq, writes to the message log, and only then acks the sender with a 202.
  • The message log is a wide-column store keyed by (conversation_id, seq), which turns both sync and scrollback into a range scan on one partition.
  • Fan-out is on write and bounded by the product's group cap, going to devices rather than users, with disconnected devices falling through to APNs or FCM.
  • Delivery and read state are cursors on the Participant row, so a 400-message backlog is one write. Be precise about what that buys: cursors collapse the per-MESSAGE dimension, not the per-PARTICIPANT one, so a 250-person group is still 250 cursor advances per message, which is why large groups batch receipts or drop per-member delivery state entirely.
  • Presence is a TTL'd key written by heartbeats and published narrowly, deliberately kept off the message path and treated as disposable.
  • Every client recovers the same way: notice a gap in seq or a newer last_seq in the conversation list, ask for the range after its cursor, and apply idempotently.

What is Expected at Each Level

  • Mid-level candidates are expected to produce the connection layer, the message store, and the fan-out, and to know that the recipient's socket lives on another machine so something has to route to it.
  • Senior candidates are expected to:
    • make the send idempotent with a client-supplied id, since without it a retry after a lost ack stores the message twice
    • put a server-assigned per-conversation seq at the centre and use it for ordering, gap detection, and sync
    • separate the per-device sync cursor from the per-user read cursor
    • ack only after the durable write, and explain why the ack ordering matters
    • model receipts as cursors instead of a row per message per recipient
    • on each deep dive, give at least the Good answer:
      • Never losing or duplicating a message: keep unsent messages in a client outbox and retry until the server acks, and make the retry idempotent with the client-supplied clientMessageId, since a lost ack otherwise stores the message twice.
      • Ordering: have the server assign a seq per conversation instead of trusting client clocks, and name the gap left when the seq is allocated in a separate step from the write.
      • Multi-device sync: keep a sync cursor per device and a read cursor per user, and name the stale advance that moves read state backwards.
      • What melts first: give reconnecting clients jittered backoff, let nodes shed load, and return sync in bounded pages, and name the silent stall when pushes stop while the socket stays up.
      • Offline delivery: treat the notification as a nudge carrying the conversation id and seq, and sync from the cursor rather than trusting the payload.
  • Staff candidates are expected to:
    • open with the concurrency math, 750M sockets and the node count that implies, and let it drive the split of the connection fleet from the service fleet
    • argue fan-out on write here against the usual social-feed answer, using the product's bounded group size as the reason
    • treat real-time push as an optimization over a pull-based sync protocol, which is what makes dropped pushes survivable
    • identify the reconnect storm rather than the database as the thing that melts, and give the boring fixes
    • name what the design does not solve: end-to-end encryption would invalidate server-side fan-out and history, and say what it would cost
DONE WHEN: you can point at the final diagram and name, in one breath, the seq, the client id, the session registry, the two cursors, and push-as-optimization.