← all topics

Design a Game Currency Wallet (Transfers + Analytics)

Hello Interview structure · plain notes, one idea per line

Understanding the Problem

💰 What are we building?

Functional Requirements

  1. Players should be able to transfer currency to each other: exactly once, never overdrawing, and safely retryable.
  2. The economy should mint and destroy currency through quests and shops, inside budgets that someone owns.
  3. Players should be able to read balances and analytics: 24-hour flows and a 30-day balance series, with stated freshness.
  4. Corruption and exploits should be detected in seconds, and the economy can be frozen while money is put back.

Below the line (out of scope):

Non-Functional Requirements

Do the scale math first, because it decides what you are allowed to build
  1. No double-spend and no negative player balance, enforced inside the database, never by app-side check-then-act. Transfers are never approximate, and amounts are integers only.
  2. The slowest 1% of transfers still complete in under 150ms (p99) at peak.
  3. Analytics may lag by seconds and must say so: every derived read carries an asOf timestamp. The truth path and the read models share no fate.
  4. Corruption is detected in seconds and a freeze lever exists. The nightly full recompute is the backstop, not the front line.

The Set Up

Defining the Core Entities

API or System Interface

POST /v1/transfers        Idempotency-Key: key   // scoped (actor, key)
  { to, amount }          // sender = auth session, never the body
  → 201 COMPLETED | 202 PENDING   // 202 = cross-shard or queued; poll the GET
  → 422, body: { error: INSUFFICIENT_FUNDS | ACCOUNT_CLOSED | ACCOUNT_FROZEN | FAUCET_BUDGET }
  → 409, body: { error: KEY_REUSED }   // same key, different body
GET  /v1/transfers/{id}                         // got a 202? ask here until COMPLETED or REJECTED
GET  /v1/accounts/{id}/balance                  → { balance, asOf }
GET  /v1/accounts/{id}/flows?window=24h         → { inflow, outflow, windowStart, windowEnd, asOf }
GET  /v1/accounts/{id}/balance-series?days=30   → { points: [{hourUtc, closingBalance}], asOf }

High-Level Design

1) Players can transfer currency to each other

wallet-hi1

2) The economy mints and destroys currency, inside budgets

wallet-hi2

3) Players can read balances and analytics, with honest freshness

wallet-hi3

4) Exploits are detected in seconds, and the economy can freeze

wallet-hi4

Potential Deep Dives

1) How is a transfer exactly-once, with no double-spend?

Bad Solution: app-side read-modify-write
  • Read the balance, check "≥ amount" in code, write the new balance. Two concurrent 100-gold spends of a 100-gold balance both read 100, both pass, both write. The account goes negative or money duplicates, and a retry after a timeout sends the money twice.
  • Everything else in this design exists to not be this.
Good Solution: fuse the guard into the debit statement
  • UPDATE accounts SET balance = balance - amt WHERE id = A AND balance - amt ≥ 0. The check and the act are one statement under the row lock, so no window exists. 0 rows affected means rejected.
  • Alone it is not enough: concurrent A→B and B→A still deadlock on lock order, a retry has no idempotency anchor to find, and a rejected transfer leaves no recorded outcome.
Great Solution: one transaction that anchors, locks in sorted order, guards, and commits either outcome
BEGIN
  INSERT transfers(actor, key, A, B, amt, PENDING)   -- dup (actor,key): replay stored outcome / 409
  SELECT status FROM accounts WHERE id = min(A,B) FOR UPDATE   -- two single-row locks,
  SELECT status FROM accounts WHERE id = max(A,B) FOR UPDATE   -- explicit sorted order
      -- missing | CLOSED | FROZEN -> SET status = REJECTED; COMMIT; 422
  UPDATE accounts SET balance = balance - amt
    WHERE id = A AND status = ACTIVE AND balance - amt ≥ floor
      -- players: floor 0. faucets: floor -budget. One guard stops
      -- overdraft AND runaway minting. 0 rows -> REJECTED; COMMIT; 422
  UPDATE accounts SET balance = balance + amt WHERE id = B AND status = ACTIVE
      -- rows-affected checked here too: no crediting a closed account
  INSERT 2 ledger legs (each with balance_after, per-account seq)
  SET status = COMPLETED
COMMIT
accounts  (id PK, status, balance, floor)
transfers (id, actor, idem_key, from, to, amount, status, reversal_of,
           UNIQUE(actor, idem_key))              -- the anchor
legs      (transfer_id, account_id, delta, balance_after, seq,
           UNIQUE(account_id, seq))              -- chain-check; seq allocated in the balance-moving txn
  • Say why, in three sentences: rejection commits, never rolls back, because nothing changed and the anchor row must survive for retries. Two explicit lock statements, because ORDER BY plus FOR UPDATE does not guarantee acquisition order (that is planner behavior, not a contract). The guard is fused into the debit statement, so no check-then-act window exists.
  • Probe: concurrent A→B and B→A where A's id sorts after B's? Both transactions issue the same two lock statements, B then A. The second waits at its first lock. No cycle, no deadlock, independent of money direction.
  • Probe: debit matches 0 rows? Nothing rolls back, because nothing changed. The same transaction marks the row REJECTED and commits. A same-key retry waits at the UNIQUE(actor, idem_key) index until that commit lands, then reads the one recorded outcome. There is no moment where the record does not exist.

2) What if the two accounts live on different shards?

Bad Solution: a distributed transaction (2PC) across ordinary shards
  • Two-phase commit makes every cross-shard transfer hold locks on two machines while waiting on a coordinator. A coordinator crash leaves rows locked in doubt, and p99 inherits the slowest shard on every transfer.
  • You built the hardest machinery in databases to avoid measuring whether trading partners could just share a shard.
Good Solution: buy Spanner-class SQL
  • One transaction even across shards, so deep dive 1's shape survives. Honest and often right.
  • Costs: slower commits, vendor lock-in, and the SELECT FOR UPDATE semantics are not identical to Postgres (CockroachDB adds retryable aborts), so "the code is unchanged" slightly overclaims.
Great Solution: a saga through per-shard escrow, fenced by one local outcomes table
  • Three small local transactions instead of one big one. Every shard has a parking account, ESCROW. Step 1: the sender's money moves to her shard's escrow. Step 2: the receiver is paid from his shard's escrow. Step 3: the two escrows settle later, in batches. One escrow only fills up, the other only drains; step 3 settles the difference.
  • Six legs total, and each step balances to zero on its own, so SUM(delta) = 0 holds at every commit boundary, and in-flight traffic never trips the conservation check.
  • The race: the payment worker and the timeout-cancel job can both act on the same transfer. The fence is an outcomes table on the receiver's shard that accepts one row per transfer id. The credit's claim is CREDITED, the cancel job's claim is CANCELLED, a frozen receiver gets REFUSED and no legs. Whoever's insert lands does its job; the other sees the existing row and gives up. Nothing ever needs to atomically observe another shard's row.
  • The receiver-shard transaction, written out: BEGIN; UPDATE accounts SET balance = balance + amt WHERE id = B AND status = ACTIVE. Rowcount 1 → INSERT outcomes(transfer_id, CREDITED) plus the ledger legs. Rowcount 0 → INSERT outcomes(transfer_id, REFUSED), no legs. COMMIT. Entirely local; the sweeper refunds escrow on REFUSED exactly as on timeout, and touches the sender shard only after winning that local race.
  • The client gets 202 PENDING and polls GET /transfers/{id}. One honest sentence: between step 2 and the lazy escrow settlement, the receiver can already spend the credit while the poll still says PENDING. Money moved, status lags by design; say it out loud rather than let the interviewer find it.
  • Bookkeeping: exclude escrow from circulation or in-flight money looks minted. And the outcomes table grows by one row per cross-shard transfer forever unless trimmed, so give it a TTL once rows age past the sweeper's horizon.

3) A dupe exploit starts at 2am. When do you find out?

Bad Solution: nightly reconciliation is the only check
  • The exploit runs from 2am until the batch job notices, up to a full day later. Twenty hours of duplicated gold moving through the market, then a cleanup that has to unwind thousands of downstream trades.
Good Solution: the right invariants, still on a batch clock
  • Check all three: legs sum to zero, balance_after chains hold per account, and MIN(player balance) ≥ 0. The third is the one naive designs miss, and it is not redundant: a guard-bypass bug that double-spends consistently passes the first two.
  • Better failure coverage, same detection latency. The 2am problem stands.
Great Solution: a continuous checker on the stream that already exists, holding a freeze lever
  • Point the CDC stream at a second consumer. The checker regroups legs by transfer id (the stream is keyed by account, so a transfer's legs arrive in different partitions), pairs them, and alarms if a leg stays unmatched past a few seconds. Chain checks ride the same flow.
  • Velocity caps catch what invariants cannot: a stolen account draining slowly is arithmetically consistent, and only rate limits on how fast and how much one account sends will flag it.
  • Escalation is mechanical, because 2am has no time for judgment: freeze the account, then freeze the economy at the gateway, transfers 503 while reads stay fine, then page a human with evidence. Cleanup is bulk reversals: compensating entries with reversal_of set, history never edited.
  • The nightly full recompute over hot plus cold tiers stays, as the backstop that catches anything the stream lied about.

4) The rollup crashes 3 seconds after a checkpoint. What do the numbers say?

Bad Solution: increment counters as events arrive
  • INCR into a cache or table per event. The increment is a side effect outside the processor's checkpointed state, so a crash rewinds the read position but not the counter, and the replayed events add again. The 24h flows number silently inflates on every recovery.
Good Solution: dedupe by entry id, key by account
  • Events keyed by account keep one account's history in order, and dropping duplicates by entry id makes redelivery harmless.
  • Not enough alone: if the flush is still "add this batch's subtotal", a crash between flush and checkpoint re-adds the subtotal. Dedupe protects against redelivery, not against replayed aggregation.
Great Solution: SET-from-state everywhere, and a series defined so errors cannot compound
  • The processor keeps the running total per (account, hour) in crash-safe state, and every flush writes the bucket's current TOTAL. Nothing in the derived path ever says "add"; everything says "the total is X". Crash 3 seconds after a checkpoint: state and read position rewind together, the same events rebuild the same totals, identical values overwrite themselves. At most seconds stale, never double-counted.
  • The series point is a lookup, not a recurrence: closing balance of hour h = balance_after of the last entry at or before the end of h, ordered by the per-account sequence. Idle hours carry forward; late data corrects only its own hours; an error cannot propagate into the next point.
  • The 24h window is decomposed honestly: 23 finished hours + the partial hour from processor state + the still-inside slice of the oldest hour. During a processor restart, the partial hour falls back to the last flushed bucket, and asOf tells the caller: stream availability degrades freshness, never correctness.

5) One marketplace account is in half of all transfers

Bad Solution: one fix for all three, split every hot account into sub-accounts
  • Splitting works only where no guard runs. For a hot SENDER, the no-overdraft guard needs the total across all sub-accounts atomically, which is exactly what splitting destroyed. Payouts start bouncing off one empty sub-account while the others hold plenty.
Good Solution: split by direction, because only one direction carries the guard
  • Hot receiver: split into sub-accounts, credits pick one at random, reads sum them. Credits have no guard, so nothing else changes, and freezing is still one status check on the parent.
  • Hot faucet: already solved by design. The guard is the budget floor, and the budget was split across the sub-faucets from the start.
  • The hot guarded sender remains, and it is the real case: the marketplace paying thousands of sellers from one balance.
Great Solution: a single-writer queue for the hot sender, with the batch's lock order fixed
  • Give that one account a queue: one Kafka partition, one consumer. The consumer batches N payouts into a single transaction; debits apply in queue order, each checked against the guard individually, and payouts that no longer fit are marked REJECTED in the same commit. Queued clients see 202 PENDING, the same contract as the saga.
  • Never two writers: before the normal synchronous path touches any account, it checks the account's queue-mode flag while holding that account's row lock, so the regime cannot flip mid-write.
  • The lock-order trap hiding in the obvious version: locking the hot account first and then the receivers "in sorted order" violates the global rule whenever a receiver sorts before the hot account. A concurrent transfer INTO the marketplace locks R then waits on H, while the batch holds H and wants R: deadlock (a liveness and p99 problem, not corruption). The fix is folded in: collect the batch first, then lock ALL touched rows, the hot account plus its N receivers, in one global sorted sequence before applying anything, guard included.

6) Lifecycle, failure, regions, retention (rapid fire)

Final Design

wallet-arch

What is Expected at Each Level