← all topics

Design a Game Currency Wallet (Transfers + Analytics)

Hello Interview structure · plain notes, one idea per line

Understanding the Problem

💰 What are we building?

Functional Requirements

  1. Players should be able to transfer currency to each other: exactly once, never overdrawing, and safely retryable.
  2. The economy should mint and destroy currency through quests and shops, inside budgets that someone owns.
  3. Players should be able to read balances and analytics: 24-hour flows and a 30-day balance series, with stated freshness.
  4. Corruption and exploits should be detected in seconds, and the economy can be frozen while money is put back.

Below the line (out of scope):

Non-Functional Requirements

Do the scale math first, because it decides what you are allowed to build
  1. No double-spend and no negative player balance, enforced inside the database, never by app-side check-then-act. Transfers are never approximate, and amounts are integers only.
  2. The slowest 1% of transfers still complete in under 150ms (p99) at peak.
  3. Analytics may lag by seconds and must say so: every derived read carries an asOf timestamp. The truth path and the read models share no fate.
  4. Corruption is detected in seconds and a freeze lever exists. The nightly full recompute is the backstop, not the front line.

The Set Up

Defining the Core Entities

API or System Interface

POST /v1/transfers        Idempotency-Key: key   // scoped (actor, key)
  { to, amount }          // sender = auth session, never the body
  → 201 COMPLETED | 202 PENDING   // 202 = cross-shard or queued; poll the GET
  → 422, body: { error: INSUFFICIENT_FUNDS | ACCOUNT_CLOSED | ACCOUNT_FROZEN | FAUCET_BUDGET }
  → 409, body: { error: KEY_REUSED }   // same key, different body
GET  /v1/transfers/{id}                         // got a 202? ask here until COMPLETED or REJECTED
GET  /v1/accounts/{id}/balance                  → { balance, asOf }
GET  /v1/accounts/{id}/flows?window=24h         → { inflow, outflow, windowStart, windowEnd, asOf }
GET  /v1/accounts/{id}/balance-series?days=30   → { points: [{hourUtc, closingBalance}], asOf }

High-Level Design

1) Players can transfer currency to each other

Transfer path: one ACID transaction, either outcome recorded A Player posts a transfer to the Gateway, which takes the sender from the session and forwards it to WalletService. WalletService runs one ACID transaction on LedgerDB: sorted locks, the conditional debit, two legs and the status, one commit. It answers the Player with 201, 202 or 422 plus a reason, and the outcome is recorded either way. Player Gateway auth session = senderrate-limits mutations WalletService validates the requestruns the one transaction LedgerDB (Postgres) accounts / transfers / legsappend-only, this is the truth 1. POST /v1/transfers {to, amount} sender = session,never the body 2. forward idem key scoped(actor,key) 3. one ACID transaction sorted locks +conditional debit +2 legs + status,one commit 4. answers 201 / 202 / 422 + reason the outcome is recorded either way the not-so-happy paths: conditional debit fails -> transfer row committed REJECTED, never rolled back (the idempotency record must survive for retries)same-key retry blocks on UNIQUE(actor, idem_key) until that commit, then reads the one recorded outcome* frozen or closed -> 422, nothing moved

2) The economy mints and destroys currency, inside budgets

Faucets and sinks: minting and destroying are ordinary transfers QuestService sends mint requests in batches to FaucetAccount, whose balance goes negative down to minus its budget. The faucet pays PlayerAccounts with an ordinary transfer under the same conditional debit. Players spend into SinkAccounts, and TreasuryAccount tops the faucet up on a schedule with an owner-approved transfer. QuestService event: pay all players FaucetAccount balance goes negativefloor = -budget, owner namedpre-split into sub-accounts PlayerAccounts floor = 0no overdraft ever SinkAccounts shop fees, taxes, repairs TreasuryAccount budget top-ups on a cadence 1. mint request sent in batches 2. ordinary transfer same conditional debit:balance - amt >= -budget 3. spend into a sink destroying currency,same mechanism 4. scheduled top-up owner-approved,itself a transfer circulation = -SUM(faucets) - SUM(sinks) <- the ledger IS the inflation dashboard the not-so-happy path: budget exhausted -> 422 FAUCET_BUDGET; a compromised payout service hits the budget wall, not the printing press

3) Players can read balances and analytics, with honest freshness

Reads and analytics: derived from the ledger, every answer carries asOf CDC tails LedgerDB's commit log and appends committed legs to KafkaLedgerStream, keyed by account. RollupProcessor reads the stream, deduplicates by entry id, and flushes hourly totals to HourlyFlows with SET. LedgerDB replicates asynchronously to Replicas. The Client reads balances from Replicas and flows from HourlyFlows, each answer marked with asOf. LedgerDB (primary) every commit alsolands in the WAL CDC tails the commit lognever a dual write KafkaLedgerStream (keyed by account)  leg leg leg RollupProcessor running totals(acct,hour)flush = SET,never INCR Replicas display readsmarked stale Client HourlyFlows derived, rebuildablefrom the ledger 1. commits committedrows only 2. append 3. feeds dedupe:entry id back: new entries front: consumers read 4. flush TOTALS overwrite-idempotenton any replay 5. async replication 6. GET /v1/accounts/ {id}/balance display readsonly 7. GET /v1/accounts/{id}/flows?window=24h and /balance-series; every answer carries asOf the not-so-happy paths: rollup crash -> state and read position rewind together, recompute, overwrite identical totals* partial hour served from processor state; during a restart fall back to the last flushed hour, asOf admits it* Kafka down -> CDC pauses at the WAL, alert on disk headroom

4) Exploits are detected in seconds, and the economy can freeze

Detection in seconds: the integrity checker holds the freeze lever KafkaLedgerStream, the same stream analytics reads, feeds legs to IntegrityChecker, which pairs them by transfer id and checks the invariants. On a violation it sets the FREEZE flag at the Gateway and pages OnCallHuman. NightlyRecompute re-sums the hot database and the cold tier as a backstop and pages if it finds something. KafkaLedgerStream the same streamanalytics reads  leg leg leg IntegrityChecker 1 all legs sum to zero2 balance_after chain holds3 MIN(player bal) >= 0+ velocity caps Gateway freeze flag checkedon every transfer OnCallHuman paged with evidence NightlyRecompute full re-sum, hot DB + cold tierbackstop, not the front line 1. feeds legs checker pairs themby transfer idalarm if unmatchedin seconds 2. FREEZE account, then economy503 transfers, reads ok 3. page a human cleanup = bulk reversals 4. found something? page safety net, not front line the not-so-happy paths: checker lagging -> alarm on consumer lag (freshness p99 < 5s)* false-positive freeze -> reads were never blocked, unfreeze is one flag

Potential Deep Dives

1) How is a transfer exactly-once, with no double-spend?

Bad Solution: app-side read-modify-write
  • Read the balance, check "≥ amount" in code, write the new balance. Two concurrent 100-gold spends of a 100-gold balance both read 100, both pass, both write. The account goes negative or money duplicates, and a retry after a timeout sends the money twice.
  • Everything else in this design exists to not be this.
Good Solution: make the debit conditional on the balance
  • UPDATE accounts SET balance = balance - amt WHERE id = A AND balance - amt ≥ 0. The check and the act are one statement under the row lock, so no window exists. 0 rows affected means rejected.
  • Alone it is not enough: concurrent A→B and B→A still deadlock on lock order, a retry has no idempotency anchor to find, and a rejected transfer leaves no recorded outcome.
Great Solution: one transaction that anchors, locks in sorted order, debits conditionally, and commits either outcome
BEGIN
  INSERT transfers(actor, key, A, B, amt, PENDING)   -- dup (actor,key): replay stored outcome / 409
  SELECT status FROM accounts WHERE id = min(A,B) FOR UPDATE   -- two single-row locks,
  SELECT status FROM accounts WHERE id = max(A,B) FOR UPDATE   -- explicit sorted order
      -- missing | CLOSED | FROZEN -> SET status = REJECTED; COMMIT; 422
  UPDATE accounts SET balance = balance - amt
    WHERE id = A AND status = ACTIVE AND balance - amt ≥ floor
      -- players: floor 0. faucets: floor -budget. One conditional debit stops
      -- overdraft AND runaway minting. 0 rows -> REJECTED; COMMIT; 422
  UPDATE accounts SET balance = balance + amt WHERE id = B AND status = ACTIVE
      -- rows-affected checked here too: no crediting a closed account
  INSERT 2 ledger legs (each with balance_after, per-account seq)
  SET status = COMPLETED
COMMIT
accounts  (id PK, status, balance, floor)
transfers (id, actor, idem_key, from, to, amount, status, reversal_of,
           UNIQUE(actor, idem_key))              -- the anchor
legs      (transfer_id, account_id, delta, balance_after, seq,
           UNIQUE(account_id, seq))              -- chain-check; seq allocated in the balance-moving txn
  • Say why, in three sentences: rejection commits, never rolls back, because nothing changed and the anchor row must survive for retries. Two explicit lock statements, because ORDER BY plus FOR UPDATE does not guarantee acquisition order (that is planner behavior, not a contract). The floor check sits inside the debit UPDATE, so no check-then-act window exists.
  • Probe: concurrent A→B and B→A where A's id sorts after B's? Both transactions issue the same two lock statements, B then A. The second waits at its first lock. No cycle, no deadlock, independent of money direction.
  • Probe: debit matches 0 rows? Nothing rolls back, because nothing changed. The same transaction marks the row REJECTED and commits. A same-key retry waits at the UNIQUE(actor, idem_key) index until that commit lands, then reads the one recorded outcome. There is no moment where the record does not exist.

2) What if the two accounts live on different shards?

Bad Solution: a distributed transaction (2PC) across ordinary shards
  • Two-phase commit makes every cross-shard transfer hold locks on two machines while waiting on a coordinator. A coordinator crash leaves rows locked in doubt, and p99 inherits the slowest shard on every transfer.
  • You built the hardest machinery in databases to avoid measuring whether trading partners could just share a shard.
Good Solution: buy Spanner-class SQL
  • One transaction even across shards, so deep dive 1's shape survives. Honest and often right.
  • Costs: slower commits, vendor lock-in, and the SELECT FOR UPDATE semantics are not identical to Postgres (CockroachDB adds retryable aborts), so "the code is unchanged" slightly overclaims.
Great Solution: a saga through per-shard escrow, fenced by one local outcomes table
  • Three small local transactions instead of one big one. Every shard has a parking account, ESCROW. Step 1: the sender's money moves to her shard's escrow. Step 2: the receiver is paid from his shard's escrow. Step 3: the two escrows settle later, in batches. One escrow only fills up, the other only drains; step 3 settles the difference.
  • Six legs total, and each step balances to zero on its own, so SUM(delta) = 0 holds at every commit boundary, and in-flight traffic never trips the conservation check.
  • The race: the payment worker and the timeout-cancel job can both act on the same transfer. The fence is an outcomes table on the receiver's shard that accepts one row per transfer id. The credit's claim is CREDITED, the cancel job's claim is CANCELLED, a frozen receiver gets REFUSED and no legs. Whoever's insert lands does its job; the other sees the existing row and gives up. Nothing ever needs to atomically observe another shard's row.
  • The receiver-shard transaction, written out: BEGIN; UPDATE accounts SET balance = balance + amt WHERE id = B AND status = ACTIVE. Rowcount 1 → INSERT outcomes(transfer_id, CREDITED) plus the ledger legs. Rowcount 0 → INSERT outcomes(transfer_id, REFUSED), no legs. COMMIT. Entirely local; the sweeper refunds escrow on REFUSED exactly as on timeout, and touches the sender shard only after winning that local race.
  • The client gets 202 PENDING and polls GET /v1/transfers/{id}. One honest sentence: between step 2 and the lazy escrow settlement, the receiver can already spend the credit while the poll still says PENDING. Money moved, status lags by design; say it out loud rather than let the interviewer find it.
  • Bookkeeping: exclude escrow from circulation or in-flight money looks minted. And the outcomes table grows by one row per cross-shard transfer forever unless trimmed, so give it a TTL once rows age past the sweeper's horizon.

3) A dupe exploit starts at 2am. When do you find out?

Bad Solution: nightly reconciliation is the only check
  • The exploit runs from 2am until the batch job notices, up to a full day later. Twenty hours of duplicated gold moving through the market, then a cleanup that has to unwind thousands of downstream trades.
Good Solution: the right invariants, still on a batch clock
  • Check all three: legs sum to zero, balance_after chains hold per account, and MIN(player balance) ≥ 0. The third is the one naive designs miss, and it is not redundant: a bug that bypasses the conditional debit and double-spends consistently passes the first two.
  • Better failure coverage, same detection latency. The 2am problem stands.
Great Solution: a continuous checker on the stream that already exists, holding a freeze lever
  • Point the CDC stream at a second consumer. The checker regroups legs by transfer id (the stream is keyed by account, so a transfer's legs arrive in different partitions), pairs them, and alarms if a leg stays unmatched past a few seconds. Chain checks ride the same flow.
  • Velocity caps catch what invariants cannot: a stolen account draining slowly is arithmetically consistent, and only rate limits on how fast and how much one account sends will flag it.
  • Escalation is mechanical, because 2am has no time for judgment: freeze the account, then freeze the economy at the gateway, transfers 503 while reads stay fine, then page a human with evidence. Cleanup is bulk reversals: compensating entries with reversal_of set, history never edited.
  • The nightly full recompute over hot plus cold tiers stays, as the backstop that catches anything the stream lied about.

4) The rollup crashes 3 seconds after a checkpoint. What do the numbers say?

Bad Solution: increment counters as events arrive
  • INCR into a cache or table per event. The increment is a side effect outside the processor's checkpointed state, so a crash rewinds the read position but not the counter, and the replayed events add again. The 24h flows number silently inflates on every recovery.
Good Solution: dedupe by entry id, key by account
  • Events keyed by account keep one account's history in order, and dropping duplicates by entry id makes redelivery harmless.
  • Not enough alone: if the flush is still "add this batch's subtotal", a crash between flush and checkpoint re-adds the subtotal. Dedupe protects against redelivery, not against replayed aggregation.
Great Solution: SET-from-state everywhere, and a series defined so errors cannot compound
  • The processor keeps the running total per (account, hour) in crash-safe state, and every flush writes the bucket's current TOTAL. Nothing in the derived path ever says "add"; everything says "the total is X". Crash 3 seconds after a checkpoint: state and read position rewind together, the same events rebuild the same totals, identical values overwrite themselves. At most seconds stale, never double-counted.
  • The series point is a lookup, not a recurrence: closing balance of hour h = balance_after of the last entry at or before the end of h, ordered by the per-account sequence. Idle hours carry forward; late data corrects only its own hours; an error cannot propagate into the next point.
  • The 24h window is decomposed honestly: 23 finished hours + the partial hour from processor state + the still-inside slice of the oldest hour. During a processor restart, the partial hour falls back to the last flushed bucket, and asOf tells the caller: stream availability degrades freshness, never correctness.

5) One marketplace account is in half of all transfers

Bad Solution: one fix for all three, split every hot account into sub-accounts
  • Splitting works only where no floor check runs. For a hot SENDER, the conditional debit needs the total across all sub-accounts atomically, which is exactly what splitting destroyed. Payouts start bouncing off one empty sub-account while the others hold plenty.
Good Solution: split by direction, because only one direction carries the floor check
  • Hot receiver: split into sub-accounts, credits pick one at random, reads sum them. Credits have no floor check, so nothing else changes, and freezing is still one status check on the parent.
  • Hot faucet: already solved by design. The faucet's floor is -budget, and the budget was split across the sub-faucets from the start.
  • The hot sender remains, and it is the real case: the marketplace paying thousands of sellers from one balance.
Great Solution: a single-writer queue for the hot sender, with the batch's lock order fixed
  • Give that one account a queue: one Kafka partition, one consumer. The consumer batches N payouts into a single transaction; debits apply in queue order, each checked against the floor individually, and payouts that no longer fit are marked REJECTED in the same commit. Queued clients see 202 PENDING, the same contract as the saga.
  • Never two writers: before the normal synchronous path touches any account, it checks the account's queue-mode flag while holding that account's row lock, so the regime cannot flip mid-write.
  • The lock-order trap hiding in the obvious version: locking the hot account first and then the receivers "in sorted order" violates the global rule whenever a receiver sorts before the hot account. A concurrent transfer INTO the marketplace locks R then waits on H, while the batch holds H and wants R: deadlock (a liveness and p99 problem, not corruption). The fix is folded in: collect the batch first, then lock ALL touched rows, the hot account plus its N receivers, in one global sorted sequence before applying anything, the floor check included.

6) Lifecycle, failure, regions, retention (rapid fire)

Final Design

Wallet: final architecture A Player posts a transfer through the Gateway to WalletService, which runs one ACID transaction on the sharded LedgerDB. LedgerDB replicates to Replicas for display reads, and CDC tails its WAL into KafkaLedgerStream. RollupProcessor folds the stream into HourlyFlows; IntegrityChecker pairs legs from the same stream and can FREEZE transfers at the Gateway. Player Gateway auth = senderrate limit + FREEZE flag WalletService idempotency (actor,key)runs the one transaction LedgerDB (sharded SQL) accounts / transfers / legsper-shard ESCROW account CDC tails the WAL Replicas display reads only KafkaLedgerStream   leg leg leg RollupProcessor SET totals, never INCRdedupe by entry id HourlyFlows serves /flows with asOf IntegrityChecker legs sum to zero * chainMIN(player bal) >= 0velocity caps * pages human 1. POST /v1/transfers sender = session,not body 2. forward if economynot frozen 3. one ACID txn sorted locks +conditional debit2 legs, 1 commit 4. replicate async, marked stale 5. WAL streams out CDC tails the WALcommitted rows, never dual-write 6. append keyed byaccount 7a. feeds 7b. flush TOTALS overwrite-idempotent 8. feeds legs the checker pairs them by transfer id, alarms in seconds 9. FREEZE transfers 503, reads ok only if asked: cross-shard saga -> per-shard escrow + outcomes fencing on the receiver shard, client polls 202 PENDING* hot sender -> one queue, one consumer, batch locks ALL touched rows in one global sorted order* DR region: async replica ledger, nothing merged * 90d hot, forever cold

What is Expected at Each Level