SAY THIS SENTENCE FIRST
"The like edge, who liked what, is the source of truth and must be exact; the count is a derived view of it that may lag a few seconds; and exactly-once lives in two places said together: the two-stage stream topology (dedupe per bucket, then regroup by item) and the fence on the counter row itself."
STEP 1 OF 5
Understanding the Problem
5 min👍 What are we building?
- Users tap like or unlike on an item (a post, a video, a comment), and everyone sees a like count on that item.
- Say the framing sentence first: the like edge, who liked what, is the source of truth and must be exact. The count is a derived view of it and may lag a few seconds. Every later decision hangs off that split.
- Three questions to ask before designing, each with a committed assumption so the interview moves even without answers:
- Is the real problem counting, or fake likes (bots, bought likes)? Assume counting, with a hook so an abuse system can retract likes later through the same path.
- Does deleting an account have to erase its likes? Assume yes, within 30 days, and the public count visibly drops. Surface that this is a product decision: a celebrity deletion visibly drains 50M likes off a public number.
- May the displayed count briefly dip or be a little stale? Assume yes: a few seconds, always labeled.
Functional Requirements
- Users should be able to like and unlike an item, and retries or duplicates must never change the count.
- Users should see an item's like count. It may lag a few seconds, but it must say so.
- The acting user should see their own like instantly, and it is always right.
- Users should be able to list who liked an item (paginated), and account deletion must erase likes with the public count dropping honestly.
Below the line (out of scope):
- Deciding which likes are fake. The abuse system owns that; this design gives it a retraction path through the same write path users take.
- Ranking, trending, and notifications. They consume the same stream this design produces; worth one sentence if asked.
- Reaction types beyond like/unlike. Same machinery, more edge states.
Non-Functional Requirements
Do the scale math first, because it picks the architecture
- 5B like events a day is about 58K/s average. Peak factor 5-10x means planning for 290-580K/s.
- Reads are 10-100x writes: 0.6-5.8M page renders a second want a count, and feed pages want N counts per render.
- About 1011 stored edges, which is tens of TB with overhead and replication.
- The one number that shapes the data model: a viral item taking 1M likes a minute is ~17K writes/s on ONE item.
- The write tier costs about $400K/month at list prices, because each action is two writes (the edge write plus the retry-protection write). At $2.5-5M a year the number does real work: it pressures build-vs-buy, and self-hosted ScyllaDB is the runner-up this bill has to beat.
- The edge is exact. Retries, duplicates, crashes, and replays never change what the count converges to.
- The count is eventually consistent with a stated bound: staleness p99 under 5 seconds, measured at the sink from event time, not guessed, and labeled
asOf. - Like-write availability 99.95%, p99 under 150ms.
- Read-your-writes for the actor (their own button), eventual for everyone else.
DONE WHEN: interviewer nods at the three assumptions, and the 17K/s-on-ONE-item number has landed.
STEP 2 OF 5
Entities + API
4 minDefining the Core Entities
- LikeEdge is one row per (item, user). It has a state, LIKED or UNLIKED, instead of being created and deleted; this row is the truth. Each real flip bumps a per-row version number (op_id) atomically in the same write.
- Count is one number per item, computed FROM the edges. Derived, never the truth.
- RetractionProposal is the abuse hook: the abuse system flips edges through the same conditional-write path as users, so retracting fake likes needs no special machinery.
API or System Interface
POST /v1/items/{itemId}/likes // Idempotency-Key: one key per tap, kept across retries
DELETE /v1/items/{itemId}/likes/{userId} // same key rules; userId must match the session
→ 200 either way // liking twice, or unliking nothing, is fine by design
GET /v1/items/{itemId}/count // → { count, asOf } - approximate, labeled
GET /v1/items/{itemId}/likes/{userId} // → { liked } - read-your-writes for the actor, else eventual
GET /v1/items/{itemId}/likes?cursor=&limit= // who liked, paginated
POST /v1/likes:batchGet { itemIds: [...] } // feed pages: N counts + N button states in one call
- The acting user comes from the login session, never from the request body.
- Same key + same request means the server replays its first answer and changes nothing.
- Like and unlike return 200 on a no-op because they set target states, not events. Asking again for a state you already have is harmless, which is exactly what makes retries safe.
- The button read gives the actor read-your-writes through the session overlay, and everyone else looking at the same row gets eventual consistency.
DONE WHEN: edge vs count restated while pointing at the entities, and the 200-on-a-no-op rule said out loud.
STEP 3 OF 5
High-Level Design
10 min, end to end, no dives yet1) Users can like and unlike, and retries never change the count
- A like is one conditional write on its edge row, which is the source of truth: set LIKED only if currently UNLIKED (or the row is new). A retry finds the state already LIKED, changes nothing, and still gets its 200.
- The database enforces one row per (item, user), so a second like can update that row but never create a second one.
- The API's Idempotency-Key is accepted but not yet used: a late retry of an old tap is where this strains; the dive on retries and duplicates handles it.
- On every successful like, the service also writes a small per-user session cache entry ("you liked X just now"), which is what makes the actor's own button instant in the read path below.
- The gateway owns auth and rate limits, and is where the abuse system's retraction proposals enter, using the same write path as users.
2) Users see a count that lags seconds, and says so
- A streaming pipeline maintains the count, because at 5B events a day it cannot be computed per read; the read returns
asOfso the seconds of lag are labeled. The same stream is what the abuse and trending systems will feed on. - The edge database publishes every committed row change, in order per row (DynamoDB Streams, NEW_AND_OLD_IMAGES). A small relay, the tailer, tails that change feed and produces each committed flip into Kafka; the stream's 24h retention bounds how long the tailer may be down.
- A stream processor reads the flips from Kafka and applies each one to the item's row in the counter store: +1 for a like, -1 for an unlike. The row holds the count and its
asOf. - One viral item sends ~17K flips a second at a single counter row; the dive on a viral post handles it.
- A processor crash replays flips it already applied, and a plain increment counts them twice; the exactly-once dive handles it.
3) The actor sees their own like instantly
TRIGGER: "millions of reads per second" / "the user taps like and sees what?"
- The actor's own button and the public count use two separate read paths, and they never mix.
- The own button is exact: it reads the per-user session-cache entry written on every like, layered over an ordinary edge read, so the tapping user sees the flip instantly even while the public count lags.
- The count is cheap: page loads read CDN cache (a few seconds of TTL) → count service → Redis → counter store. Most reads never reach a database.
- When caches expire together on a viral item, serve the stale copy while one refresh per item runs (stale-while-revalidate plus request coalescing).
- Feed pages make one collection-level batch call,
POST /v1/likes:batchGetwith the page's item ids, that returns N counts plus N button states together.
4) Who liked an item, and erasure that drops the count honestly
- Who liked an item: page through that item's LIKED edge rows with a cursor, one page per call.
- Edge rows grouped by item put one viral item's ~17K writes a second on one partition; the dive on a viral post handles it.
- Erasure: deleting an account hard-deletes its edge rows AND its retry-protection rows (the stored answers behind its Idempotency-Keys). Each deletion flows through the pipeline as a minus-one, so counts drop honestly and publicly.
- Kafka retention (7 days) is shorter than the 30-day erasure window, so a rebuild replay can never re-materialize erased data.
DONE WHEN: the interviewer picks a box to open. Let them steer from here.
- If they just nod and wait: "the riskiest part is making the count exactly right under retries, shall I open that?"
- Running behind: protect the retries dive and the exactly-once dive. Cut the multi-region dive first.
STEP 4 OF 5
Potential Deep Dives
~20 min, interviewer steersTRIGGER: "how do retries not double-count" / "walk me through a like"
1) How do retries and duplicates never change the count?
- The trap has three layers, and each protects against a different failure. The one that kills naive designs: a like's 200 is lost, the user unlikes, and the client retries the OLD like.
Bad Solution: dedupe requests in application memory
- The service remembers recent request ids in memory and drops repeats. A restart forgets everything; a retry that lands on another node was never seen.
- Worse, any read-then-write ("is it liked? no? then like it") is check-then-act: two concurrent taps both read UNLIKED and both apply.
Good Solution: put idempotency in the data model
- Layer 1, the row: one row per (item, user), enforced unique by the database. A second like updates that row; it can never create a second row.
- Layer 2, the conditional write: set state to LIKED only if it is currently UNLIKED (or the row is new). A retry finds LIKED, changes nothing, still gets 200. Each real flip bumps op_id atomically in the same write (DynamoDB:
if_not_exists(op_id,0)+1inside the conditional UpdateItem; a no-op emits no stream record). - This makes double-likes harmless, but it cannot tell a stale retry from a genuine new intent.
Great Solution: add the request key, and name the hole it closes
- Each layer stops one duplicate: a unique row stops a second row, a conditional write a repeated state change, a request key a stale retry.
- Layer 3 is the request key: each tap gets a key, and retries of that tap reuse it, and the server replays the stored answer for (user, key), kept about a day, without touching state.
- The hole it closes: like → 200 lost → user unlikes → the old retry arrives. State checks alone see UNLIKED and would re-apply the like, flipping the row against the user's last choice. The key makes that retry replay its original 200 instead; the row stays UNLIKED.
TRIGGER: "one post goes viral" / "what melts first"
2) One post goes viral. What melts first?
- Steelman the simpler design first: edge store + a per-event ADD on the counter row with op_id as the dedupe token, no Kafka, no stream processor. That is where to start at small scale, and honesty about where it breaks is the point: a storage partition caps near 1K writes/s, so it breaks at the first modestly viral event, not the mega-viral one.
Bad Solution: partition the edge table by item_id and defend everything downstream
- The viral item's 17K conditional writes/s land on ONE partition that takes about 1K/s. The source of truth melts before Kafka or the counter ever see load.
- Defending the stream and the counter while the truth store dies is defending the wrong tier.
Good Solution: bucket the truth store's key
- Row key item + hash(user) % 64: one viral item's writes spread over 64 partitions, ~260/s each. Deterministic by user, so (item, user) uniqueness and per-edge ordering survive.
- Name the cost out loud: the full who-liked scan now fans out to 64 buckets (rare path), and the recount job inherits the same fan-out.
- Acceptable, because that full 64-bucket fan-out stays an admin and recount path, not a user path.
- The user-facing who-liked page is not free either: a page of likers has no single-bucket home, so serving one page still means a multi-bucket read, a cost this design accepts out loud.
Great Solution: one lever per tier, and the keying doubles as the stream's dedupe locality
- Each tier gets one lever, and the edge store's is the bucket key: row key item + hash(user) % 64 spreads one item over 64 partitions.
- Kafka is keyed by the SAME item#bucket rule, so a retried event meets its own dedupe memory.
- The key is
item#hash(user)%64, the same deterministic rule everywhere. Deterministic matters: a retried event lands in the same group and meets the duplicate-check that already saw it (the dedupe stage remembers the last op_id applied per edge and drops anything at or below it). A random spread would dodge it and double-count.
- The key is
- The counter row sees one write per item per second because windowed aggregation adds each second up first, far under any per-row limit.
- The per-item window step adds the deltas up one second at a time, so "+8,214" replaces 8,214 writes.
- If a single counter row ever needs sharding, its fence (the "only if this second is newer" check) must shard with it deterministically: route windows by
shard = window_id % K, never randomly, or a replayed window can land on a different shard and pass that shard's fence. One sentence, but it is the difference between sharded and broken. - The tuning dial to watch is per-bucket write throttles: a throttle firing means the armor is breached and the bucket count needs to grow for that keyspace.
TRIGGER: "exactly once?" / "the processor crashes" / "why not count in the stream"
3) Exactly once: crash the processor mid-window and tell me the count
- This is the probe the whole design exists to survive. The answer must locate the guarantee at the LAST write, not in the stream.
Bad Solution: count in the stream processor and INCRBY the store
- Input dedupe protects the processor's input; nothing protects its output. After a crash, state and offsets restore together, the same windows are rebuilt, and the increments already applied to the counter apply a second time.
- "Offsets plus dedupe give effectively-once" attributes the guarantee to the wrong mechanism.
Good Solution: idempotent sink via a guard table keyed (item, window)
- Right instinct, two real defects. The guard write and the counter write are separate tables: a crash between them either loses a window or double-applies it, which violates the design's own "never check-then-act" maxim at its most important write.
- And the guard table grows by ~109 rows a day unless it gets a TTL at least as long as the replay horizon.
Great Solution: the fence lives on the counter row itself
UPDATE counter SET count = count + :n, last_window = :w CONDITION last_window < :w -- one atomic statement; a replayed window fails the condition and does nothing
- The guarantee lives at the last write: one atomic update on the counter row adds the second's total and sets last_window (the newest second the row has applied), only if last_window is older than this second (the fence). Guard and increment are one atomic update on one row, the side table is deleted from the schema, and everything upstream may replay freely.
- Stage 1 drops duplicates: each dedupe worker owns some bucket keys and remembers the last op_id (the edge row's version number) applied per edge, dropping any event at or below it. One viral item's 64 bucket keys land on 64 dedupe workers.
- Stage 2 regroups by item: a keyBy(item) shuffle brings all 64 buckets' deltas for an item to one window subtask, which emits ONE (item, second, +N) per second.
- Why the regroup is stated: without it, 64 workers each write partial windows, and any per-item guard at the sink accepts the first partial and wrongly rejects the other 63. That is a 63/64 undercount on exactly the viral case.
- The one-second viral trace, end to end: 16.7K events land across 64 dedupe workers; the keyBy(item) regroup brings that item's deltas to ONE worker; it emits exactly one (item, second, +N); the sink is one atomic update with the fence on the counter row, so there is no gap between check and add. Crash anywhere: the replayed second re-presents, fails the fence, the count stays right.
- The crash trace in one breath: the processor's memory and reading position rewind together to the last checkpoint, it re-reads the same events, rebuilds the same per-second totals, re-offers them, and each replayed second bounces off the fence. The count is at most seconds stale and never double-counted.
- The tailer (the relay from the change feed into Kafka) needs an idempotent producer,
enable.idempotence=true(ordering preserved with up to 5 in flight); there is no per-key in-flight knob. Dedupe memory is TTL'd to the 7-day replay window, not all 1011 edges, with reconciliation named as the backstop past it.
TRIGGER: "counter store dies" / "counts drift" / "a celebrity deletes their account"
4) The counter store dies. Counts drift. A celebrity deletes their account.
- Three separate repair scenarios, each self-contained, all leaning on one property: the counter is derived, so it can always be rebuilt from the edges.
Bad Solution: recount the edges and overwrite the counter
- The recount reflects time T; the overwrite lands at T+delta and destroys every delta applied during the scan, worst on hot items, which the job prioritizes. Reconciliation becomes the design's best corruption vector.
Good Solution: rebuild from the truth, replay the bounded tail
- Counter store lost: recount the edges in parallel, then replay the last 7 days of stream on top. Full-history replay does not exist; the stream keeps 7 days, and the edge store is the rebuild source.
- Take the recount from a frozen snapshot AT the replay start position, and initialize each rebuilt counter's fence at that same position, so the replayed windows apply exactly once. A live recount with the fence set 7 days back would count the tail twice.
Great Solution: fenced corrections through the pipeline, never overwrites
- The counter is derived from the edges, so every repair rebuilds from them and sends the difference through the pipeline as a correction; nothing ever overwrites the counter.
- If the counter store dies, recount the edges from a frozen snapshot and replay the last 7 days of stream on top.
- Drift: recount an item from a frozen snapshot of the edges (a point-in-time recovery export at stream position T), wait until the sink watermark passes T, then send the difference as a correction through the normal pipeline. The correction enters the stage-2 keyed stream as its own window, so the row fence serializes it with live traffic instead of overwriting it.
- Clamp at zero and alert: a negative count is a bug report, not a number.
- Celebrity deletion needs no repair machinery at all: 50M unlikes flow as ordinary minus-ones, spread across buckets, rate-limited. The erasure job is just a big polite user.
TRIGGER: "multi-region?" / "a region dies"
5) Multi-region, honestly
- The commitment must be stated before the failover story: who is allowed to write an edge, and what happens to that authority when a region dies mid-write.
Bad Solution: active-active replication of the edge store
- Two regions writing the same edge resolve by last-writer-wins, which silently breaks the exactness the whole design is built on. Cross-region conditional writes are impractical; global tables are LWW.
Good Solution: one home region per item range
- Each item range has one home region that owns its writes; other regions forward writes there (priced: far users pay a cross-region hop on writes) and serve reads locally, labeled stale.
- Reads never need the home region; only the write path does.
Great Solution: epoch-fenced pin moves, with the on-call signal named
- Each item range has one home region that owns its writes, and it carries an epoch (era number) that every conditional write must match.
- To move a range, bump the epoch FIRST at the new home, so the old home's in-flight writes fail closed instead of forking history. They carry a stale epoch.
- The unreplicated tail from the old home becomes a reconciliation correction through the pipeline, never a merge.
- What on-call sees, in order: the old-home rejection spike first, then traffic shifting to the new home. That ordering is the health check for the move itself.
- Where the epoch check lives is still an open choice. A conditional write evaluates against the row being written, so either every edge row carries the epoch attribute (with a bump-on-first-write migration story) or the writer checks an out-of-band range lease, and that check must not reintroduce check-then-act. The fencing-token shape is right; commit to a storage location when asked.
TRIGGER: "how do you run this" / "what do you watch"
6) How do you run it? SLOs, alerts, backpressure, and the cutover
- Running it means measured targets, alerts mapped to them, a degradation order under load, and a way to change the pipeline without double-counting.
Bad Solution: guess at staleness, queue writes, switch pipelines in place
- Staleness is inferred rather than measured, so nobody can say whether the 5-second target holds.
- Overloaded like-writes queue up instead of failing fast with retry-after.
- A new pipeline switched on over the old one lets the old one's tail double-apply on a live counter.
Good Solution: measured SLOs with alerts mapped to them
- Targets: like-write availability 99.95% with p99 under 150ms; count staleness p99 under 5s, measured as watermark lag from event time at the sink, not inferred.
- Alerts map to those targets: sink watermark lag (staleness burning), consumer lag, per-bucket write throttles (the hot-item armor is breached), reconciliation mismatch (any nonzero pages), retry-protection error rate.
- On-call now sees a promise breaking, but nothing yet says what gives first under load or how to switch pipelines without double-counting.
Great Solution: a degradation order that keeps every promise, and a cutover at a marked stream position
- Every promise the requirements made holds under load and during a pipeline change: writes fail fast, counts go stale rather than wrong, and a cutover cannot double-count.
- Targets: like-write availability 99.95% with p99 under 150ms; count staleness p99 under 5s, measured as watermark lag from event time at the sink, not inferred.
- Alerts map to those targets: sink watermark lag (staleness burning), consumer lag, per-bucket write throttles (the hot-item armor is breached), reconciliation mismatch (any nonzero pages), retry-protection error rate.
- When overloaded: fail like-writes fast with retry-after, never queue them; reads keep serving. Counts go visibly stale rather than wrong, and own-buttons stay correct via the session overlay.
- Cutover: roll out a new pipeline by running old and new side by side in shadow and diffing daily, then cut over at a marked stream position by setting each counter row's last-applied second (last_window) there, so the new pipeline's replays bounce and the old one's tail cannot double-apply. Rollback is the same move at another mark. This is the dedupe-regime switch done on a live counter without double-counting.
- Runbooks cover counter rebuild (recount + bounded replay), home-region move (epoch increment as step one), bucket-count growth for a melting keyspace.
DONE WHEN: every opened dive reached its Great ending, and the probes below were answered from memory before the reveal.
FLASHCARDS
The five hardest probes
show all (interview mode)One second of the viral item, end to end.
16.7K events land across 64 dedupe workers. The regroup step brings all their deltas for that item to ONE worker. It emits exactly one (item, second, +N). The sink is one atomic update with the fence "only if this second is newer" on the counter row itself, so there is no gap between check and add. Crash anywhere: the replayed second re-presents, fails the fence, the count stays right.
Why must the Kafka grouping be a hash of the user, never random?
Because the duplicate-check memory lives with the group. A deterministic rule sends a retried event to the same worker that already saw it. A random spread sends it to a fresh worker with no memory of it, which applies it again: guaranteed double count on exactly the retry paths that matter.
The processor crashes 3 seconds after its last checkpoint. What does the count read?
Its memory and reading position rewind together. It re-reads the same events, rebuilds the same per-second totals, and re-offers them to the counter. Each replayed second fails the counter's fence and does nothing. The count is at most seconds stale and never double-counted.
A like's 200 is lost, the user unlikes, the client retries the old like.
The retry carries the original key. The server finds (user, key) in its retry-protection table and replays the stored 200 without touching state. The row stays UNLIKED, matching what the user last chose. State checks alone would have re-applied the like; the key is what makes it safe.
Home region for a hot range goes dark mid-write. Move it without forking history.
Bump the range's era number first, at the new home. The old home's in-flight conditional writes now carry a stale era and fail closed. Then serve writes at the new home. The old home's unreplicated tail is reconciled later as corrections through the pipeline, never merged. On-call sees the old-home rejection spike first, then traffic shift.
STEP 5 OF 5
Final Design + What is Expected at Each Level
3 minFinal Design
- A like is one conditional write on its edge row, the source of truth, with the request key replaying lost answers and the session overlay making the actor's button instant.
- The change feed streams every committed flip into Kafka keyed item#hash(user)%64. Stage 1 drops duplicates per bucket; the keyBy(item) shuffle brings an item's deltas to one window subtask; it emits one (item, second, +N).
- The counter row's fence (ADD + SET last_window, CONDITION last_window < :w) makes every replay a no-op. The guarantee lives at the last write.
- Product pages read through CDN, count service, and Redis; the count is labeled asOf and may lag seconds. The two read paths never mix.
- Reconciliation recounts from a snapshot and heals through the same pipeline as its own fenced window. Erasure flows as rate-limited minus-ones. Region moves bump the range epoch first so stragglers fail closed.
What is Expected at Each Level
- Mid-level candidates are expected to state the edge/count split, get one row per (item, user) with a conditional write, cache the read path, and know the count is maintained by a pipeline rather than computed per read.
- Senior candidates are expected to carry idempotency through all three layers (including the stale-retry-after-unlike hole the request key closes), state the two-stage stream topology with deterministic keying and explain the 63/64 undercount the missing shuffle would cause, put the fence on the counter row itself, bucket the truth store with the cost named, and give the bounded rebuild story (edge recount plus 7-day replay). For multi-region, a senior answer gives each item range one home region that owns its writes, with other regions forwarding writes there and serving reads locally, labeled stale. For operations, it states the targets (like-write 99.95% with p99 under 150ms, count staleness p99 under 5s measured at the sink) and maps alerts to them.
- Staff candidates are expected to interrogate the requirements before designing (counting vs integrity, erasure as a visible product decision), do the scale math that finds the viral item's 17K/s against the 1K/s partition cap, steelman the simpler design and name its real breakpoint, let the $400K/month write bill pressure build-vs-buy, bring the operations story unprompted (SLOs measured not inferred, alerts mapped to them, a degradation order that keeps the requirements' promises, the shadow cutover that switches dedupe regimes safely), fence the region move with epochs and admit where the epoch's storage is still an open choice, and show restraint: present requirements through high-level design in about 20 minutes and let the interviewer pick the deep dives.
DONE WHEN: the five-bullet final walkthrough fits in one breath each, and you know which dive carries the staff signal.
NEXT ACTION, 3 MINUTES
Read the five Final Design bullets aloud once, timing yourself. If they run past 90 seconds, cut words until they fit. Then answer probe 1 from memory before opening it.