← all topics Wallet board · say it in this order 1 Require 4m 2 Entities 1m 3 API 3m 4 Design 10m 5 Dives ~20m reset
SAY THIS SENTENCE FIRST

"Every movement of currency, including minting and destroying it, is a transfer between accounts in one append-only double-entry ledger. Everything else is derived from it."

STEP 1 OF 5

Requirements

4 min
ASK, THEN COMMIT

Four questions. State your assumption for each so the interview moves even without answers.

  1. Player-to-player only, or do quests and shops mint and destroy?  Assume mints: system accounts, first-class.
  2. Duplication exploit at 2am: what response?  Assume detection in seconds plus a freeze lever.
  3. GM refunds and bug cleanups exist?  Assume yes: compensating entries.
  4. Multi-region?  Assume explicit cut: one region of record plus DR replica.
Numbers to say

100M transfers/day ≈ 1.2K/s avg · 10-15K/s peak · reads 20-50x writes · slowest 1% of transfers still < 150ms (p99) · ~200M legs/day ≈ 20GB/day, TB-scale/yr · integers only

Hard rules

No double-spend. No negative player balance. Analytics may lag seconds and must say so. Transfers are never approximate.

DONE WHEN: interviewer nods at the four assumptions.
STEP 2 OF 5

Core entities

1 min
  • Account: holds a balance. Player accounts may never go below zero. System accounts (quest faucets, shop sinks) may, within a budget someone owns.
  • Transfer: one requested movement, with a lifecycle.
  • Ledger entry: every transfer writes two entries, called legs: minus on the sender, plus on the receiver. They always sum to zero.
  • Rollups: per-account, per-hour summaries computed FROM the ledger. Derived, never the truth.
DONE WHEN: you have said the opening sentence again, pointing at these four nouns. Schema waits for dive A.
STEP 3 OF 5

API

3 min
POST /v1/transfers        Idempotency-Key: key   // scoped (actor,key)
  { to, amount }          sender = auth session, never the body
  → 201 COMPLETED | 202 PENDING (cross-shard / queued)
  → 422, body: { error: INSUFFICIENT_FUNDS | ACCOUNT_CLOSED | ACCOUNT_FROZEN | FAUCET_BUDGET }
  → 409, body: { error: KEY_REUSED }   // same key, different body
GET  /v1/transfers/{id}   got a 202? ask here repeatedly until COMPLETED or REJECTED
GET  /v1/accounts/{id}/balance              → { balance, asOf }
GET  /v1/accounts/{id}/flows?window=24h     → { inflow, outflow, windowStart, windowEnd, asOf }
GET  /v1/accounts/{id}/balance-series?days=30 → { points:[{hourUtc, closingBalance}], asOf }
  • Same key + same body replays the stored outcome. Retries are exact, not accidental.
  • 202 PENDING is a first-class state, shared by the saga and the hot-sender queue.
  • Validated: amount > 0, to ≠ actor, amount ≤ cap.
  • Gateway rate-limits mutations; analytics reads are authorized to the account owner or a service principal. Every derived read carries asOf: freshness is part of the contract.
STEP 4 OF 5

High-level design

10 min, end to end, no dives yet
Wallet architecture: transfer path, propagation with freeze lever, read paths
Whiteboard version: 6 boxes, this draw order
client → gateway → wallet svc → LEDGER (truth)          -- row 1, left to right
LEDGER ↓ CDC/Kafka → rollup → counts   +   checker      -- row 2
checker ↑↑ FREEZE back to gateway (dashed red)           -- the one arrow that makes the argument
everything else (replicas, reconciliation, escrow) only if asked
THE WALKTHROUGH, ONE BREATH PER CLAUSE

Every movement is one ACID transaction in a sharded SQL ledger → cross-shard: dive D: two sorted row locks, one fused guard (the balance check lives inside the debit UPDATE), two legs, status committed together → dive A. CDC streams committed changes into Kafka feeding two consumers: idempotent rollups → dive E and a continuous integrity checker with a freeze lever → dive G. Spending reads hit the primary; display reads hit replicas, staleness marked. Nightly recompute is the backstop.

DONE WHEN: the interviewer picks a box to open. Let them steer from here.
  • If they just nod and wait: "the riskiest box is the transfer transaction, shall I open it?"
  • Running behind: protect dives A and G. Cut C first.
  • If asked "why Kafka at this scale": one Postgres primary plus a one-minute SQL rollup is the right first system. The streaming tier was bought for one thing: catching corruption in seconds.
STEP 5 OF 5

Deep dives: the core five

~20 min, interviewer steers
TRIGGER: "walk me through a transfer" / "how do you prevent double-spend"
A · The transfer transaction
BEGIN
  INSERT transfers(actor, key, A, B, amt, PENDING)   -- dup(actor,key): replay / 409
  SELECT status FROM accounts WHERE id = min(A,B) FOR UPDATE   -- two single-row locks,
  SELECT status FROM accounts WHERE id = max(A,B) FOR UPDATE   -- explicit sorted order
      -- missing | CLOSED | FROZEN → status=REJECTED; COMMIT; 422
  UPDATE accounts SET balance = balance - amt
    WHERE id = A AND status = ACTIVE AND balance - amt ≥ floor
      -- players: floor 0. faucets: floor -budget. One guard stops
      -- overdraft AND runaway minting. 0 rows → REJECTED; COMMIT; 422
  UPDATE accounts SET balance = balance + amt WHERE id = B AND status = ACTIVE
  INSERT 2 ledger legs (each with balance_after, per-account seq)
  SET status = COMPLETED
COMMIT

accounts  (id PK, status, balance, floor)
transfers (id, actor, idem_key, from, to, amount, status, reversal_of,
           UNIQUE(actor, idem_key))              -- the anchor
legs      (transfer_id, account_id, delta, balance_after, seq,
           UNIQUE(account_id, seq))              -- chain-check; seq allocated in the balance-moving txn

Say why, in three sentences: Rejection commits, never rolls back; nothing changed, and the anchor row must survive for retries. Two explicit lock statements because ORDER BY + FOR UPDATE does not guarantee acquisition order. The guard is fused into the debit statement, so no check-then-act window exists.

TRIGGER: "what if the two accounts are on different shards"
D · Cross-shard transfers
ANSWER 1 OF 2
Start here
  • Spanner-class SQL: one transaction, even across shards.
  • Dive A's code stays the same.
  • Cost: slower commits, vendor lock-in.
ANSWER 2 OF 2 · everything indented below is part of this one
Otherwise: the saga
  • Three small local transactions, not one big one.
  • Every shard has a parking account: ESCROW.
  • Step 1: sender's money → her shard's escrow.
  • Step 2: receiver paid from his shard's escrow.
  • Step 3: square the two escrows later, in batches.
  • Six legs; each step balances to zero on its own.
Slow credit vs cancel: only one may win
  • Two workers can end up acting on the same transfer at once: the payment step, and the cancel job.
  • If both act, the receiver gets paid AND the sender gets refunded. Money doubled.
  • So: before touching any money, each worker must first write its claim into one table (outcomes) on the receiver's shard.
  • The payment's claim is the row CREDITED. The cancel job's claim is the row CANCELLED.
  • The table accepts only one row per transfer, so only one claim can ever land.
  • The worker whose claim landed does its job. The other sees the existing row and gives up.
  • All of this happens on one shard, so no machine ever waits on another.
Frozen receiver
  • The payment step finds the receiver's account frozen or closed.
  • It writes REFUSED as its claim and moves no money.
  • The cancel job sees REFUSED and refunds the sender from escrow, exactly like a timeout.
The cancel job
  • Step 1 also leaves a note: "transfer X started at 12:00:00."
  • A background job scans for notes older than the timeout.
  • When it finds one, it tries to cancel, through the claim table above.
Bookkeeping
  • One escrow only fills up, the other only drains; step 3 settles the difference between them.
  • Exclude escrow from circulation, or in-flight money looks minted.
Client
  • Gets 202 ("accepted, still working"). Asks GET /transfers/{id} until the status settles.
Say at the end
  • Do not build this until cross-shard pairs are measured as common.
  • Same-shard placement of trading partners may defer it forever.
TRIGGER: "a quest pays every player 100 gold" / anything about the economy
C · Faucets, sinks, inflation
ONE ANSWER IN 4 PARTS · all the same mechanism: system accounts
Minting
  • A quest reward = normal transfer, faucet account → player.
  • The faucet's floor is its budget.
  • Same guard, two jobs: stops overdrafts, caps minting.
Destroying
  • Shop fees and taxes = transfers into sink accounts.
Inflation dashboard, free
  • Money in circulation = minus the faucet balances, minus the sink balances.
  • Faucet balances are negative, so the result comes out positive.
  • The ledger is the dashboard. No separate metrics system.
The quest event
  • Paying every player = N ordinary transfers, sent in batches.
  • The faucet is split into many sub-accounts ahead of time, so the burst never hammers one row.
  • Afterward the ledger still sums to exactly zero.
TRIGGER: "2am dupe exploit" / "how fast do you detect corruption"
G · Integrity, continuous
ONE SYSTEM · what it checks, how fast, what it does about it
Three invariants
  • 1. All legs sum to zero: nothing created or destroyed.
  • 2. Every ledger entry records the balance after it. Each entry must equal the previous balance plus its own change. Any gap = corruption.
  • 3. No player balance is ever negative.
  • The third is not redundant: a bug that double-spends the same coins consistently passes checks 1 and 2, and only check 3 catches it.
Velocity caps
  • Limits on how fast and how much one account may send.
  • Catches abuse the balance guard cannot see: stolen accounts draining slowly, bots.
Checking conservation
  • Events are grouped by account, so a transfer's two legs end up in two different groups.
  • So a separate checker regroups events by transfer id and pairs each transfer's two legs.
  • Alarm if a leg stays unmatched past a few seconds.
Speed
  • The checker reads the same event stream as analytics, so it reacts in seconds instead of waiting for the nightly job.
Escalation
  • Freeze the account.
  • Then freeze the economy at the gateway: transfers 503, reads fine.
  • Then page a human. Cleanup = bulk reversals.
Backstop
  • Nightly full recompute (hot DB + cold tier). Safety net, not front line.
TRIGGER: "analytics" / "how fresh are the numbers" / "rollup crash"
E · Analytics, idempotent and self-healing
ONE PIPELINE · read top to bottom
The feed
  • CDC (change data capture) watches the ledger database and copies every committed row into Kafka.
  • Events are keyed by account, so one account's history stays in order.
  • The rollup drops duplicates by entry id.
Why replays cannot double-count
  • The processor keeps a running total per (account, hour) in its own crash-safe storage.
  • Flush writes the bucket's current TOTAL. Never an increment.
  • Crash → recompute the same totals → overwrite with identical values.
The series point
  • Hour's close = balance_after of the last entry at or before hour end.
  • Idle hour: carry the previous value forward.
  • Late data corrects only the hours it touches. Nothing compounds.
24h flows, concretely
  • Table: hourly_flows(account, hour, inflow, outflow).
  • Answer = the sum of the last 24 hours, ending right now.
  • That is 23 finished hours, plus the current unfinished hour, plus whatever slice of the oldest hour still falls inside the window.
The partial hour
  • Served from the processor's own state.
  • During restart: fall back to last flushed hour; asOf admits it.
STEP 5, CONT.

Three more dives

rehearse after the core four
TRIGGER: "close an account with money in it" / "who can refund" / "crash leaves PENDING rows?"
B · Lifecycle: closure, reversals, PENDING
4 SEPARATE TOPICS · each answers a different question
Closing an account with money in it
  • The remainder transfers to a system escheat account.
  • Conservation holds right through closure.
Reversals
  • A refund is a NEW transfer in the opposite direction, with its reversal_of field pointing at the original.
  • Never an edit to history.
  • Only a dedicated service identity, two-person approval, own audit log.
  • It is the one path allowed to move money its caller does not own.
Two different PENDINGs
  • The transfer row's internal PENDING: born and finished inside one transaction. Nobody outside ever sees it.
  • The API's 202 PENDING: cross-shard or queued transfers, resolved by poll.
The sweeper's whole job
  • Saga intent rows that waited too long. Nothing else needs sweeping.
TRIGGER: "marketplace account" / "one account gets all the traffic"
F · Hot accounts, both directions
THREE DIFFERENT CASES, EACH WITH ITS OWN FIX
CASE 1
Hot receiver
  • Split into sub-accounts. Credits pick one at random; reads sum them.
  • Credits have no guard, so nothing else changes.
  • Freezing it = one status check on the parent.
CASE 2
Hot faucet
  • Already handled: the guard is the budget floor, split across shards.
CASE 3 · everything indented below is part of this one
Hot guarded sender (marketplace)
  • Give that one account a queue: one Kafka partition, one consumer.
  • The consumer batches N payouts into a single transaction.
Inside the batch
  • Debits apply in queue order, each checked individually.
  • Payouts that do not fit get their transfer rows marked REJECTED, in the same commit.
  • Queued clients see 202 PENDING.
Never two writers
  • Before the normal path touches an account, it checks whether that account is in queue mode.
  • The check happens while holding the account's row lock, so the answer cannot change mid-write.
Locking
  • The batch locks every account it touches in global sorted order.
TRIGGER: "region goes down" / "Kafka dies" / "how long do you keep the ledger"
H · Failure, regions, retention
6 SEPARATE SCENARIOS · each self-contained
Multi-region: the cut, said out loud
  • One region owns all writes. A second region keeps a copy, always slightly behind, for disaster recovery.
  • True region loss: at most seconds lost, reconciled from the replica ledger.
  • Nothing is ever merged.
Kafka down
  • CDC pauses at the database's own commit log.
  • The real limit: how long the database's own log (the WAL) can keep growing. Alert on the space left.
  • Worst case: rebuild derived stores from the ledger.
Primary down
  • Reject transfers immediately. Never queue money you cannot commit.
  • Balances stay readable from replicas, marked stale.
Retention
  • 90 days in the fast database; everything, forever, in cheap cold storage.
  • The ledger gets point-in-time backups: restore to any minute.
  • Analytics store: derived, no backup needed.
Numbers I watch
  • Transfer p99 + success rate; rejections by reason.
  • Analytics freshness p99 < 5s; consumer lag.
  • Reconciliation mismatches = 0; replication lag; hot-queue depth.
Simplest design
  • At 1.2K/s: one Postgres primary + one-minute SQL rollup. Start there.
  • The streaming tier buys one thing: corruption caught in seconds, not overnight.
FLASHCARDS

The five hardest probes

show all (interview mode)
Concurrent A→B and B→A, A's id sorts after B's?
Both transactions issue the same two lock statements, B then A. Second one waits at its first lock. No cycle, no deadlock, independent of money direction.
Debit matches 0 rows?
Nothing rolls back, because nothing changed. The same transaction marks the transfer row REJECTED and commits. A retry with the same key waits at the database's uniqueness check until that commit lands, then reads the one recorded outcome. There is no moment where the record does not exist.
Quest pays every player 100 gold: what does SUM(delta) read after?
Exactly zero. Faucet shards go 100·N more negative within budget, players +100 each, every transfer's legs sum to zero. The event's mint total IS the faucet shards' aggregate delta: the inflation dashboard updating itself.
Rollup crashes 3s after checkpoint: what does /flows return?
The processor's memory and its reading position rewind together to the last checkpoint. It re-reads the same events, rebuilds the same running totals, and writes those totals again, overwriting identical values. Nothing in the derived path ever says "add"; everything says "the total is X." So the number is at most seconds stale, and never double-counted.
Write me the receiver-shard saga transaction.
BEGIN; UPDATE accounts SET balance = balance + amt WHERE id = B AND status = ACTIVE. Rowcount 1 → INSERT outcomes(transfer_id, CREDITED) + ledger legs. Rowcount 0 (receiver frozen or closed mid-saga) → INSERT outcomes(transfer_id, REFUSED), no legs. COMMIT. Entirely local. CREDITED, REFUSED, and the sweeper's CANCELLED are mutually exclusive on the primary key; the sweeper refunds escrow on REFUSED exactly as on timeout, and touches the sender shard only after winning that local race.