← all topics Like counter board · say it in this order 1 Problem 2 Entities + API 3 High-level design 4 Deep dives 5 Final + levels reset
SAY THIS SENTENCE FIRST

"The like edge, who liked what, is the source of truth and must be exact; the count is a derived view of it that may lag a few seconds; and exactly-once lives in two places said together: the two-stage stream topology (dedupe per bucket, then regroup by item) and the fence on the counter row itself."

STEP 1 OF 5

Understanding the Problem

5 min
👍 What are we building?
  • Users tap like or unlike on an item (a post, a video, a comment), and everyone sees a like count on that item.
  • Say the framing sentence first: the like edge, who liked what, is the source of truth and must be exact. The count is a derived view of it and may lag a few seconds. Every later decision hangs off that split.
  • Three questions to ask before designing, each with a committed assumption so the interview moves even without answers:
  1. Is the real problem counting, or fake likes (bots, bought likes)? Assume counting, with a hook so an abuse system can retract likes later through the same path.
  2. Does deleting an account have to erase its likes? Assume yes, within 30 days, and the public count visibly drops. Surface that this is a product decision: a celebrity deletion visibly drains 50M likes off a public number.
  3. May the displayed count briefly dip or be a little stale? Assume yes: a few seconds, always labeled.

Functional Requirements

  1. Users should be able to like and unlike an item, and retries or duplicates must never change the count.
  2. Users should see an item's like count. It may lag a few seconds, but it must say so.
  3. The acting user should see their own like instantly, and it is always right.
  4. Users should be able to list who liked an item (paginated), and account deletion must erase likes with the public count dropping honestly.

Below the line (out of scope):

  • Deciding which likes are fake. The abuse system owns that; this design gives it a retraction path through the same write path users take.
  • Ranking, trending, and notifications. They consume the same stream this design produces; worth one sentence if asked.
  • Reaction types beyond like/unlike. Same machinery, more edge states.

Non-Functional Requirements

Do the scale math first, because it picks the architecture
  • 5B like events a day is about 58K/s average. Peak factor 5-10x means planning for 290-580K/s.
  • Reads are 10-100x writes: 0.6-5.8M page renders a second want a count, and feed pages want N counts per render.
  • About 1011 stored edges, which is tens of TB with overhead and replication.
  • The one number that shapes the data model: a viral item taking 1M likes a minute is ~17K writes/s on ONE item.
  • The write tier costs about $400K/month at list prices, because each action is two writes (the edge write plus the retry-protection write). At $2.5-5M a year the number does real work: it pressures build-vs-buy, and self-hosted ScyllaDB is the runner-up this bill has to beat.
  1. The edge is exact. Retries, duplicates, crashes, and replays never change what the count converges to.
  2. The count is eventually consistent with a stated bound: staleness p99 under 5 seconds, measured at the sink from event time, not guessed, and labeled asOf.
  3. Like-write availability 99.95%, p99 under 150ms. Under overload, fail writes fast with retry-after rather than queueing them; reads keep serving.
  4. Read-your-writes for the actor (their own button), eventual for everyone else.
DONE WHEN: interviewer nods at the three assumptions, and the 17K/s-on-ONE-item number has landed.
STEP 2 OF 5

Entities + API

4 min

Defining the Core Entities

  • LikeEdge is one row per (item, user). It has a state, LIKED or UNLIKED, instead of being created and deleted; this row is the truth. Each real flip bumps a per-row version number (op_id) atomically in the same write. The row key is item + hash(user) % 64, so one item's rows spread over 64 partitions.
  • Count is one number per item, computed FROM the edges. Derived, never the truth. The counter row also remembers last_window, the newest second it has applied; that field is the dedupe fence.
  • IdempotencyRecord maps (user, request key) to the stored answer for about a day, so a retried request replays its first answer.
  • RetractionProposal is the abuse hook: the abuse system flips edges through the same conditional-write path as users, so retracting fake likes needs no special machinery.

API or System Interface

POST   /v1/items/{itemId}/likes            // Idempotency-Key: one key per tap, kept across retries
DELETE /v1/items/{itemId}/likes/{userId}   // same key rules; userId must match the session
  → 200 either way                        // liking twice, or unliking nothing, is fine by design
GET    /v1/items/{itemId}/count            // → { count, asOf } - approximate, labeled
GET    /v1/items/{itemId}/likes/{userId}   // → { liked } - read-your-writes for the actor, else eventual
GET    /v1/items/{itemId}/likes?cursor=&limit=   // who liked, paginated
POST   /v1/likes:batchGet   { itemIds: [...] }   // feed pages: N counts + N button states in one call
  • The acting user comes from the login session, never from the request body.
  • Same key + same request means the server replays its first answer and changes nothing.
  • Why 200 on a no-op: like and unlike are target states, not events. Asking again for a state you already have is harmless, which is exactly what makes retries safe.
  • The button read is honest about its guarantee: read-your-writes for the actor via the session overlay, eventual for anyone else looking at the same row.
DONE WHEN: edge vs count restated while pointing at the entities, and the 200-on-a-no-op rule said out loud.
STEP 3 OF 5

High-Level Design

10 min, end to end, no dives yet

1) Users can like and unlike, and retries never change the count

lc-hi1
  • A like is one conditional write on its edge row, which is the source of truth: set LIKED only if currently UNLIKED (or the row is new). A retry finds the state already LIKED, changes nothing, and still gets its 200.
  • The database enforces one row per (item, user), so a second like can update that row but never create a second one.
  • The request key closes the gap state checks cannot: the service keeps (user, key) → answer for about a day and replays the stored answer. Deep dive 1 walks the exact failure this exists for.
  • On every successful like, the service also writes a small per-user session cache entry ("you liked X just now"), which is what makes the actor's own button instant in HLD-3.
  • The gateway owns auth and rate limits, and is where the abuse system's retraction proposals enter, using the same write path as users.

2) Users see a count that lags seconds, and says so

lc-hi2
  • Why streaming at all: at 5B events a day the count cannot be computed per read; something must maintain it. The same stream is what the abuse and trending systems will feed on.
  • The edge database publishes every committed row change, in order per row (DynamoDB Streams, NEW_AND_OLD_IMAGES). A small relay, the tailer, tails that change feed and produces each committed flip into Kafka; the stream's 24h retention bounds how long the tailer may be down.
  • Events land in Kafka keyed item#hash(user)%64, the same deterministic rule everywhere. Deterministic matters: a retried event lands in the same group and meets the duplicate-check that already saw it. A random spread would dodge it and double-count.
  • Stage 1: dedupe workers partitioned by bucket key; for one viral item's traffic, its 64 bucket keys land on 64 dedupe workers, each remembering the last op_id applied per edge. An event at or below that number has been seen: drop it. This memory only covers the 7-day replay window, not all 1011 edges; reconciliation is the backstop beyond it.
  • Stage 2, and it must be SAID as a stage: a keyBy(item) shuffle brings all 64 buckets' deltas for an item back together at one window subtask, which adds them up one second at a time and emits ONE (item, second, +N): "+8,214" instead of 8,214 writes.
  • Why the regroup is stated: without it, 64 workers each write partial windows, and any per-item guard at the sink accepts the first partial and wrongly rejects the other 63. That is a 63/64 undercount on exactly the viral case.
  • The counter row itself carries the fence: ADD the total and SET last_window, but only if this second is newer. One atomic statement; a replayed second fails the condition and does nothing. Deep dive 3 stress-tests this.

3) The actor sees their own like instantly

TRIGGER: "millions of reads per second" / "the user taps like and sees what?"
lc-hi3
  • Two separate read paths, and they never mix.
  • The count, cheap: page loads read CDN cache (a few seconds of TTL) → count service → Redis → counter store. Most reads never reach a database.
  • Two standard brakes against the viral-item stampede when caches expire together: serve the stale copy while refreshing, and let only one refresh per item through at a time.
  • The own button, exact: button state is the session-cache overlay layered over an ordinary edge read, so the tapping user sees the flip instantly even while the public count lags.
  • Feed pages make one collection-level batch call, POST /v1/likes:batchGet with the page's item ids, that returns N counts plus N button states together.

4) Who liked an item, and erasure that drops the count honestly

lc-hi4
  • The bucket key earns its place here and in deep dive 2: a single partition takes about 1K writes/s, and the viral item does ~17K/s. Without buckets the source of truth melts first and nothing downstream matters.
  • What the bucket costs: "show me every liker of this item" now has to ask all 64 buckets. Acceptable, because that full 64-bucket fan-out stays an admin and recount path, not a user path. The user-facing who-liked page is not free either: a page of likers has no single-bucket home, so serving one page still means a multi-bucket read, a cost this design accepts out loud.
  • Erasure: deleting an account hard-deletes its edge rows AND its retry-protection rows. Each deletion flows through the pipeline as a minus-one, so counts drop honestly and publicly.
  • Celebrity-scale deletion (50M unlikes) is just a big polite user: ordinary minus-ones, spread across buckets, rate-limited. No special path.
  • Stream retention (7 days) is shorter than the 30-day erasure window, so a rebuild replay can never re-materialize erased data.
DONE WHEN: the interviewer picks a box to open. Let them steer from here.
  • If they just nod and wait: "the riskiest part is making the count exactly right under retries, shall I open that?"
  • Running behind: protect dives 1 and 3. Cut dive 5 (regions) first.
STEP 4 OF 5

Potential Deep Dives

~20 min, interviewer steers
TRIGGER: "how do retries not double-count" / "walk me through a like"

1) How do retries and duplicates never change the count?

  • The trap has three layers, and each protects against a different failure. The one that kills naive designs: a like's 200 is lost, the user unlikes, and the client retries the OLD like.
Bad Solution: dedupe requests in application memory
  • The service remembers recent request ids in memory and drops repeats. A restart forgets everything; a retry that lands on another node was never seen.
  • Worse, any read-then-write ("is it liked? no? then like it") is check-then-act: two concurrent taps both read UNLIKED and both apply.
Good Solution: put idempotency in the data model
  • Layer 1, the row: one row per (item, user), enforced unique by the database. A second like updates that row; it can never create a second row.
  • Layer 2, the conditional write: set state to LIKED only if it is currently UNLIKED (or the row is new). A retry finds LIKED, changes nothing, still gets 200. Each real flip bumps op_id atomically in the same write (DynamoDB: if_not_exists(op_id,0)+1 inside the conditional UpdateItem; a no-op emits no stream record).
  • This makes double-likes harmless, but it cannot tell a stale retry from a genuine new intent.
Great Solution: add the request key, and name the hole it closes
  • Layer 3: each tap gets a key; retries of that tap reuse it. The server keeps (user, key) → answer for about a day and replays the stored answer without touching state.
  • The hole it closes: like → 200 lost → user unlikes → the old retry arrives. State checks alone see UNLIKED and would re-apply the like, flipping the row against the user's last choice. The key makes that retry replay its original 200 instead; the row stays UNLIKED.
  • Say where each layer's guarantee lives: the row kills duplicates of existence, the conditional write kills duplicates of transition, the key kills duplicates of intent.
TRIGGER: "one post goes viral" / "what melts first"

2) One post goes viral. What melts first?

  • Steelman the simpler design first: edge store + a per-event ADD on the counter row with op_id as the dedupe token, no Kafka, no stream processor. That is where to start at small scale, and honesty about where it breaks is the point: a storage partition caps near 1K writes/s, so it breaks at the first modestly viral event, not the mega-viral one.
Bad Solution: partition the edge table by item_id and defend everything downstream
  • The viral item's 17K conditional writes/s land on ONE partition that takes about 1K/s. The source of truth melts before Kafka or the counter ever see load.
  • Defending the stream and the counter while the truth store dies is defending the wrong tier.
Good Solution: bucket the truth store's key
  • Row key item + hash(user) % 64: one viral item's writes spread over 64 partitions, ~260/s each. Deterministic by user, so (item, user) uniqueness and per-edge ordering survive.
  • Name the cost out loud: the full who-liked scan now fans out to 64 buckets (rare path), and the recount job inherits the same fan-out.
Great Solution: one lever per tier, and the keying doubles as the stream's dedupe locality
  • Truth store: the bucket key (above). Stream: the SAME item#bucket rule keys Kafka, so a retried event meets its own dedupe memory. Counter: windowed aggregation means the counter row sees one write per item per second, far under any per-row limit.
  • If a single counter row ever needs sharding anyway, the fence must shard with it deterministically: route windows by shard = window_id % K, never randomly, or a replayed window can land on a different shard and pass that shard's fence. One sentence, but it is the difference between sharded and broken.
  • The tuning dial to watch is per-bucket write throttles: a throttle firing means the armor is breached and the bucket count needs to grow for that keyspace.
TRIGGER: "exactly once?" / "the processor crashes" / "why not count in the stream"

3) Exactly once: crash the processor mid-window and tell me the count

  • This is the probe the whole design exists to survive. The answer must locate the guarantee at the LAST write, not in the stream.
Bad Solution: count in the stream processor and INCRBY the store
  • Input dedupe protects the processor's input; nothing protects its output. After a crash, state and offsets restore together, the same windows are rebuilt, and the increments already applied to the counter apply a second time.
  • "Offsets plus dedupe give effectively-once" attributes the guarantee to the wrong mechanism.
Good Solution: idempotent sink via a guard table keyed (item, window)
  • Right instinct, two real defects. The guard write and the counter write are separate tables: a crash between them either loses a window or double-applies it, which violates the design's own "never check-then-act" maxim at its most important write.
  • And the guard table grows by ~109 rows a day unless it gets a TTL at least as long as the replay horizon.
Great Solution: the fence lives on the counter row itself
UPDATE counter SET
  count = count + :n,
  last_window = :w
CONDITION last_window < :w
-- one atomic statement; a replayed window fails the condition and does nothing
  • Guard and increment are one atomic update on one row; the side table is deleted from the schema. The guarantee lives at the last write, so everything upstream may replay freely.
  • The one-second viral trace, end to end: 16.7K events land across 64 dedupe workers; the keyBy(item) regroup brings that item's deltas to ONE worker; it emits exactly one (item, second, +N); the sink is one atomic update with the fence on the counter row, so there is no gap between check and add. Crash anywhere: the replayed second re-presents, fails the fence, the count stays right.
  • The crash trace in one breath: the processor's memory and reading position rewind together to the last checkpoint, it re-reads the same events, rebuilds the same per-second totals, re-offers them, and each replayed second bounces off the fence. The count is at most seconds stale and never double-counted.
  • Kafka honesty: the tailer's producer needs enable.idempotence=true (ordering preserved with up to 5 in flight); there is no per-key in-flight knob. Dedupe memory is TTL'd to the 7-day replay window, with reconciliation named as the backstop past it.
TRIGGER: "counter store dies" / "counts drift" / "a celebrity deletes their account"

4) The counter store dies. Counts drift. A celebrity deletes their account.

  • Three separate repair scenarios, each self-contained, all leaning on one property: the counter is derived, so it can always be rebuilt from the edges.
Bad Solution: recount the edges and overwrite the counter
  • The recount reflects time T; the overwrite lands at T+delta and destroys every delta applied during the scan, worst on hot items, which the job prioritizes. Reconciliation becomes the design's best corruption vector.
Good Solution: rebuild from the truth, replay the bounded tail
  • Counter store lost: recount the edges in parallel, then replay the last 7 days of stream on top. Full-history replay does not exist; the stream keeps 7 days, and the edge store is the rebuild source.
  • Take the recount from a frozen snapshot AT the replay start position, and initialize each rebuilt counter's fence at that same position, so the replayed windows apply exactly once. A live recount with the fence set 7 days back would count the tail twice.
Great Solution: fenced corrections through the pipeline, never overwrites
  • Drift: recount an item from a frozen snapshot of the edges (PITR/snapshot export at stream position T), wait until the sink watermark passes T, then send the difference as a correction through the normal pipeline. The correction enters the stage-2 keyed stream as its own window, so the row fence serializes it with live traffic instead of overwriting it.
  • Clamp at zero and alert: a negative count is a bug report, not a number.
  • Celebrity deletion needs no repair machinery at all: 50M unlikes flow as ordinary rate-limited minus-ones. The erasure job is just a big polite user.
TRIGGER: "multi-region?" / "a region dies"

5) Multi-region, honestly

  • The commitment must be stated before the failover story: who is allowed to write an edge, and what happens to that authority when a region dies mid-write.
Bad Solution: active-active replication of the edge store
  • Two regions writing the same edge resolve by last-writer-wins, which silently breaks the exactness the whole design is built on. Cross-region conditional writes are impractical; global tables are LWW.
Good Solution: one home region per item range
  • Each item range has one home region that owns its writes; other regions forward writes there (priced: far users pay a cross-region hop on writes) and serve reads locally, labeled stale.
  • Reads never need the home region; only the write path does.
Great Solution: epoch-fenced pin moves, with the on-call signal named
  • Every range carries an era number (epoch), and every conditional write requires the current era. To move a range: bump the era FIRST at the new home. The old home's in-flight writes now carry a stale era and fail closed instead of forking history.
  • The unreplicated tail from the old home becomes a reconciliation correction through the pipeline, never a merge.
  • What on-call sees, in order: the old-home rejection spike first, then traffic shifting to the new home. That ordering is the health check for the move itself.
  • One honest open point, stated rather than hidden: where the epoch check lives. A conditional write evaluates against the row being written, so either every edge row carries the epoch attribute (with a bump-on-first-write migration story) or the writer checks an out-of-band range lease, and that check must not reintroduce check-then-act. The fencing-token shape is right; commit to a storage location when asked.
TRIGGER: "how do you run this" / "what do you watch"

6) How do you run it? SLOs, alerts, backpressure, and the cutover

  • Targets: like-write availability 99.95% with p99 under 150ms; count staleness p99 under 5s, measured as watermark lag from event time at the sink, not inferred.
  • Alerts map one-to-one to those targets: sink watermark lag (staleness burning), consumer lag, per-bucket write throttles (the hot-item armor is breached), reconciliation mismatch (any nonzero pages), retry-protection error rate.
  • When overloaded: fail like-writes fast with retry-after, never queue them; reads keep serving. Counts go visibly stale rather than wrong, and own-buttons stay correct via the session overlay. The degradation order is a choice, and this one keeps every promise the requirements made.
  • Rolling out a new pipeline: run old and new side by side in shadow and diff daily; cut over at a marked stream position by initializing each counter's fence there, so the new pipeline's replays bounce and the old one's tail cannot double-apply. Rollback is the same move at another mark. This is the dedupe-regime switch done on a live counter without double-counting.
  • Runbook heads: counter rebuild (recount + bounded replay), pin move (epoch increment as step one), bucket-count growth for a melting keyspace.
DONE WHEN: every opened dive reached its Great ending, and the probes below were answered from memory before the reveal.
FLASHCARDS

The five hardest probes

show all (interview mode)
One second of the viral item, end to end.
16.7K events land across 64 dedupe workers. The regroup step brings all their deltas for that item to ONE worker. It emits exactly one (item, second, +N). The sink is one atomic update with the fence "only if this second is newer" on the counter row itself, so there is no gap between check and add. Crash anywhere: the replayed second re-presents, fails the fence, the count stays right.
Why must the Kafka grouping be a hash of the user, never random?
Because the duplicate-check memory lives with the group. A deterministic rule sends a retried event to the same worker that already saw it. A random spread sends it to a fresh worker with no memory of it, which applies it again: guaranteed double count on exactly the retry paths that matter.
The processor crashes 3 seconds after its last checkpoint. What does the count read?
Its memory and reading position rewind together. It re-reads the same events, rebuilds the same per-second totals, and re-offers them to the counter. Each replayed second fails the counter's fence and does nothing. The count is at most seconds stale and never double-counted.
A like's 200 is lost, the user unlikes, the client retries the old like.
The retry carries the original key. The server finds (user, key) in its retry-protection table and replays the stored 200 without touching state. The row stays UNLIKED, matching what the user last chose. State checks alone would have re-applied the like; the key is what makes it safe.
Home region for a hot range goes dark mid-write. Move it without forking history.
Bump the range's era number first, at the new home. The old home's in-flight conditional writes now carry a stale era and fail closed. Then serve writes at the new home. The old home's unreplicated tail is reconciled later as corrections through the pipeline, never merged. On-call sees the old-home rejection spike first, then traffic shift.
STEP 5 OF 5

Final Design + What is Expected at Each Level

3 min

Final Design

lc-arch
  • A like is one conditional write on its edge row, the source of truth, with the request key replaying lost answers and the session overlay making the actor's button instant.
  • The change feed streams every committed flip into Kafka keyed item#hash(user)%64. Stage 1 drops duplicates per bucket; the keyBy(item) shuffle brings an item's deltas to one window subtask; it emits one (item, second, +N).
  • The counter row's fence (ADD + SET last_window, CONDITION last_window < :w) makes every replay a no-op. The guarantee lives at the last write.
  • Product pages read through CDN, count service, and Redis; the count is labeled asOf and may lag seconds. The two read paths never mix.
  • Reconciliation recounts from a snapshot and heals through the same pipeline as its own fenced window. Erasure flows as rate-limited minus-ones. Region moves bump the range epoch first so stragglers fail closed.

What is Expected at Each Level

  • Mid-level candidates are expected to state the edge/count split, get one row per (item, user) with a conditional write, cache the read path, and know the count is maintained by a pipeline rather than computed per read.
  • Senior candidates are expected to carry idempotency through all three layers (including the stale-retry-after-unlike hole the request key closes), state the two-stage stream topology with deterministic keying and explain the 63/64 undercount the missing shuffle would cause, put the fence on the counter row itself, bucket the truth store with the cost named, and give the bounded rebuild story (edge recount plus 7-day replay).
  • Staff candidates are expected to interrogate the requirements before designing (counting vs integrity, erasure as a visible product decision), do the scale math that finds the viral item's 17K/s against the 1K/s partition cap, steelman the simpler design and name its real breakpoint, let the $400K/month write bill pressure build-vs-buy, bring the operations story unprompted (SLOs measured not inferred, alerts mapped to them, a degradation order that keeps the requirements' promises, the shadow cutover that switches dedupe regimes safely), fence the region move with epochs and admit where the epoch's storage is still an open choice, and show restraint: present requirements through high-level design in about 20 minutes and let the interviewer pick the deep dives.
DONE WHEN: the five-bullet final walkthrough fits in one breath each, and you know which dive carries the staff signal.