# Wallet transfers + analytics board: panel review

Artifact v1: `wallet-transfers-analytics_2.html` / v2: `wallet-transfers-analytics-v2.html`

---

# V3 REVIEW (2026-08-08): 4.75 / 5 - strong hire at both levels

Method: line diff against v2, each fix verified against the v2 verification findings, plus my own composition hunt on the new mechanisms. All 14 v2-round findings are fixed, several with genuinely elegant unifications:

- N1 saga blocker -> per-shard escrow, SIX legs with every transaction locally zero-sum (verified: each of the three transactions sums to zero, so SUM(delta)=0 holds at every commit boundary), and fencing moved entirely onto the receiver shard: an `outcomes` table where the credit's CREDITED insert and the sweeper's CANCELLED tombstone are mutually exclusive on the primary key. "Nothing ever needs to atomically observe another shard's row." The sweeper touches the sender shard only after winning the local race. Receiver-side idempotency falls out of the same insert. The verifier's live probe is answered verbatim in the worked-probes section, with the receiver-shard transaction written out.
- N2 -> FROZEN checked in step 2 AND `status = ACTIVE` in both the debit and credit statements, with ACCOUNT_FROZEN in the 422 surface. Belt and suspenders.
- N3 -> two single-row SELECT FOR UPDATE statements in explicit sorted sequence, with the planner-dependency reason stated on the board.
- N7 -> the standout fix: a `floor` column unifies everything - players floor 0 (no overdraft), faucets floor -budget with a named owner. One fused guard `balance - amt >= floor` stops both player overdrafts and runaway minting; a compromised payout service "hits the budget wall instead of printing forever." 422 FAUCET_BUDGET; budgets split across sub-faucet shards.
- N5/N6 -> 202 PENDING is a first-class API state with a poll endpoint; same-key retry replays 202 with current status; queued hot-sender transfers use the same contract; regime flips can't create two writers because the hot flag is checked under the account's row lock; batch semantics defined (apply in queue order until the guard fails, reject remainder individually).
- N8 -> per-account sequence number assigned in-transaction; hour bucket from a timestamp written in the same statement; "no mixed clocks."
- N9 -> privileged reversal path explicit: distinct service principal, dual authorization, own audit records, named as the most abuse-sensitive surface.
- N10-N14 -> per-shard escrow; balance_after chain-checking with periodic spot-compare; nightly recompute against OLTP + tiered cold ledger; partial-hour degrades to last flushed bucket with honest asOf ("stream availability for freshness, never for correctness"); PENDING-never-outlives-the-transaction justification corrected; Spanner/CockroachDB claim softened accurately.

## Residual findings (my composition hunt)

1. MINOR (the one real new issue) - The hot-sender batch locks the hot account FIRST, then credits receivers "in sorted id order." That violates the global sorted-lock rule whenever a receiver sorts before the hot account: a concurrent synchronous transfer R -> H (someone paying the marketplace) locks R then waits on H, while the batch holds H and wants R - deadlock (detector kills one; liveness/p99 issue, not corruption, and confined to hot-account + counterparty pairs). Fix: collect the batch, lock ALL rows (hot account + N receivers) in one global sorted sequence before applying, guard included.
2. NIT - Between saga step 2 and the lazy step 3 settlement, the receiver can spend the credit while GET /transfers/{id} still returns PENDING. Money moved, status lags. Acceptable, worth one honest sentence.
3. NIT - `outcomes` table retention unstated (one row per cross-shard transfer, forever); TTL past the sweeper horizon.
4. NIT - Faucet budget replenishment cadence unstated (budgets deplete monotonically; the named owner presumably tops up, but the mechanism/cadence isn't on the board).

## Scorecard trajectory

| Dimension | Wt | v1 | v2 | v3 |
|---|---|---|---|---|
| Requirements and estimation | 10% | 3.0 | 5.0 | 5.0 |
| API design | 10% | 2.5 | 4.0 | 5.0 |
| Data model | 10% | 3.0 | 4.5 | 4.5 |
| Core architecture | 15% | 3.5 | 4.5 | 5.0 |
| Correctness guarantees | 20% | 2.0 | 3.5 | 4.5 |
| Scale and hot items | 10% | 3.0 | 4.0 | 4.5 |
| Failure and recovery | 10% | 3.0 | 4.5 | 5.0 |
| Ops, cost, evolution | 10% | 3.0 | 4.5 | 4.5 |
| Communication / artifact | 5% | 3.0 | 5.0 | 5.0 |
| **Weighted** | | **2.8** | **4.3** | **4.75** |

## Verdict

Strong hire at senior and staff. The seams that held v2 at "hire" are closed with mechanisms that survived adversarial reading, and two fixes (the floor unification, the outcomes-table fencing) are better than what the reviewers suggested. Remaining probe surface: the hot-batch lock order (residual 1) and the ops dimension's missing capacity/cost arithmetic (the one line this board never gained: what does the write tier cost, and what does that number decide?).

## Delivery note (same as like-counter v4)

The artifact is now complete enough that the risk flips to restraint: present Requirements -> Entities -> API -> High-Level in ~20 minutes, let the interviewer choose deep dives, and resist reciting A-H unprompted. The open-questions section is your best asset in the room - lead with it when asked "what would you do differently."

---

# V2 REVIEW (2026-08-08): 4.3 / 5 - hire at senior, hire at staff

Method: I verified all 19 v1 findings against the v2 text quote-by-quote myself (all addressed, most with the exact recommended mechanism), then a senior payments agent ran a new-composition hunt on the rewrite. Score up from 2.8.

## All 19 v1 findings: fixed

Highlights of HOW (each verified in the v2 text): sorted SELECT FOR UPDATE before the debit with a worked deadlock trace; rejection COMMITS the status onto the surviving anchor ("nothing to roll back"); MIN(player balance) >= 0 as a third invariant WITH the non-redundancy argument; system accounts as the centerpiece (faucets mint by going negative, inflation dashboard = -SUM(faucet)-SUM(sink), the quest-payout probe worked end to end); continuous integrity checker on the CDC stream with account-freeze and economy-freeze levers ("point the stream at the checker first"); reversals as compensating entries with reversal_of; PENDING + escrow + intent table making the saga representable; sender = session actor by construction; (actor,key)-scoped idempotency, 409 on key reuse with different body, 422 for INSUFFICIENT_FUNDS; SET-from-state flushes with "no INCR anywhere in the derived path" and Redis removed from the write path; closing_balance(h) redefined from balance_after (kills the recurrence); hot-sender split into faucet-no-floor and single-writer-queue halves; multi-region as an explicit, defended cut; WAL retention honesty; PITR + 90d OLTP tiering; simplest-design threshold with "say which one bought it: the checker did"; coaching tone gone; open-questions section with a genuinely good product question (mint provenance vs exploit map).

## New-composition findings (senior verification round)

### N1 - BLOCKER: the saga fencing is a cross-shard CAS that cannot exist as specified
"The receiver-shard credit commits only under the same CAS" - but the transfers row lives on the sender's shard. Atomically checking a sender-shard row while committing a receiver-shard write IS a distributed transaction, the thing the saga exists to avoid, reintroduced inside the fence. Without it the v1 mint race is reborn: receiver reads PENDING, begins credit; sweeper CASes REJECTED and refunds escrow; credit commits anyway - refund + credit both land. Also: no receiver-shard idempotency record exists, so a retried step-2 message double-credits. Workable pattern: sweeper writes a cancellation tombstone ON the receiver shard first (late credit aborts locally), THEN releases escrow. Mitigation: the board itself defers the saga as unbuilt ("until cross-shard pairs are measured"), which is the right call - but it ships the deferred design with a false correctness claim.

### Significant
- N2: FROZEN is claimed "checked under lock in dive A" but dive A only rejects missing/CLOSED; the debit's WHERE tests balance and kind, not status. The per-account kill switch - the 2am fix - is unwired in the only code shown (422 lists no FROZEN code either).
- N3: ORDER BY + FOR UPDATE is not a guaranteed lock-acquisition order (InnoDB locks during scan before filesort; Postgres LockRows-above-Sort is undocumented behavior). Robust form: two single-row SELECT FOR UPDATE in sorted order. The deadlock fix rests on planner behavior.
- N4: "Four legs summing to zero at every step" is arithmetically false - between step 2 (credit) and step 3 (escrow release), global SUM(delta) = +amt. Zero-at-every-step needs 6 legs with a receiver-side escrow.
- N5: committed-PENDING has no API contract: the cross-shard original request cannot return 201 COMPLETED, and a duplicate retry mid-saga finds PENDING with "replay stored outcome" undefined. The idempotency fix and the saga fix interact to open a hole neither had alone.
- N6: the hot-sender queue reintroduces two writers (nothing routes or blocks the synchronous path during regime flips), breaks the synchronous 201 contract (no 202/PENDING defined), and leaves batch partial-shortfall semantics plus the N+1-row lock order unstated.
- N7: the faucet OR-clause makes unlimited minting invisible to all three invariants (legs sum to zero, balances consistent, faucets legitimately negative); only velocity caps would catch it, and the board's own open questions say those numbers are unset. No per-faucet floor/budget exists. The most dangerous write path is the one exempted from the guard.
- N8: closing_balance(h) mixes timelines - "last entry at or <= end(h)" evaluates created_at (assigned pre-lock) while balance_after encodes commit order; the two can disagree at hour boundaries, and the "self-healing" materialization then reproduces the same wrong answer forever. Fix: define "last" by the per-account sequence balance_after already encodes, pin hour membership to one stated timeline.

### Minor / nit
- N9: GM reversals debit accounts no GM owns - the privileged path that violates actor=sender is unstated, and it is the most abuse-sensitive path.
- N10: ESCROW is a single unsharded hot row for all cross-shard traffic (dive F pre-shards faucets, dive D forgets its own medicine); its home shard is also unstated.
- N11: invariant (2)'s incremental mechanism is underspecified (entries stream lacks accounts.balance; chain-checking balance_after is a different check), and the nightly full SUM over ALL legs collides with the 90-day OLTP window.
- N12: the partial hour couples reads to stream-processor availability with no stated fallback during restarts.
- N13: "crash recovery has a state to inspect" is false on the single-shard path (PENDING rolls back with the txn) - justification wrong, behavior right.
- N14: "dive A's code unchanged" on Spanner overclaims (no SELECT FOR UPDATE in the Postgres sense; CockroachDB retryable aborts).

## What survived attack (verified sound)
Rejection-commits path (duplicate blocks on the unique index, reads recorded outcome); fused guarded debit with no check-then-act gap under held locks; credit existence + rows-affected checks; quest-payout locking with pre-sharded faucets (SUM(delta)=0 verified); the inflation formula (verified arithmetically); SET-from-state rollup idempotence including the crash-3s-after-checkpoint trace; the fixed 24h window; the invariant-(3) non-redundancy argument; the sender-shard half of the sweeper; WAL framing; closure-as-escheat; all arithmetic.

## Scorecard

| Dimension | Wt | v1 | v2 |
|---|---|---|---|
| Requirements and estimation | 10% | 3.0 | 5.0 |
| API design | 10% | 2.5 | 4.0 |
| Data model | 10% | 3.0 | 4.5 |
| Core architecture | 15% | 3.5 | 4.5 |
| Correctness guarantees | 20% | 2.0 | 3.5 |
| Scale and hot items | 10% | 3.0 | 4.0 |
| Failure and recovery | 10% | 3.0 | 4.5 |
| Ops, cost, evolution | 10% | 3.0 | 4.5 |
| Communication / artifact | 5% | 3.0 | 5.0 |
| **Weighted** | | **2.8** | **4.3** |

## Verdict
Hire at senior (verifier's words: "strong core, unreliable seams"), hire at staff (all three staff criticals converted into the design's strongest sections). Held below strong hire by the compositional seams: the saga fence, FROZEN unwired, the lock-order planner dependency, and the faucet blind spot.

## To reach 4.5+
1. Wire FROZEN into dive A's step-2 check and the 422 surface (one line, and it is the 2am lever).
2. Two single-row SELECT FOR UPDATE statements in sorted order (kill the planner dependency).
3. Saga: receiver-shard cancellation tombstone + receiver-shard idempotency record; fix "summing to zero at every step" to the 6-leg form or drop the claim; define the 202/PENDING API contract that both the saga and the hot-sender queue need.
4. Per-faucet budgets/floors with owners, so minting has a hard bound, not just unset velocity caps.
5. Define closing_balance's timeline by per-account sequence; state the privileged reversal path; shard the escrow account.

## Remaining live probe (the verifier's)
"Your delayed receiver-shard credit 'loses the CAS and aborts.' The transfers row is on the sender's shard. Write me the receiver-shard transaction: what statement, executed on the receiver's shard, atomically observes that CAS - and if you need to read the sender's shard, what stops the sweeper from firing between your read and your commit?"

---

# V1 REVIEW (original, unchanged below)

Artifact: `wallet-transfers-analytics_2.html` (reviewed 2026-08-07)
Panel: senior payments engineer (mechanisms), staff engineer (judgment), adversarial fact-checker (truth only). Three independent agents.

## Committee verdict: 2.8 / 5 - lean hire at senior, no-hire at staff

Both graded reviewers landed on lean hire independently. The spine is right and the instincts are visibly practitioner-grade, but the money path ships two certain blockers of the "states the correct rule, then exhibits code that violates it" species, the headline invariant cannot catch the failure it advertises, and the domain (a GAME economy) is systematically missing from an otherwise excellent generic wallet.

## Scorecard

| Dimension | Wt | Score | Notes |
|---|---|---|---|
| Requirements and estimation | 10% | 3.0 | All arithmetic verified clean; interrogation performed, not done; P2P-only scoping never flagged |
| API design | 10% | 2.5 | asOf on read models is right; nothing enforces from == actor; idempotency keys unscoped; 409 vs 422 |
| Data model | 10% | 3.0 | Double-entry + balance_after is the right idea; saga unrepresentable in the schema; no mint/reversal/lifecycle |
| Core architecture | 15% | 3.5 | CDC-not-dual-write, fate isolation, derived-rebuildable all correct; component count not earned at 1.2K/s |
| Correctness guarantees | 20% | 2.0 | Two certain blockers in the money path; reconciliation blind to negatives; saga mint race |
| Scale and hot items | 10% | 3.0 | Hot receiver addressed; hot sender (the board's own example) unsolved; "24h" window is really 24-25h |
| Failure and recovery | 10% | 3.0 | Degradation ladder genuinely good; hour-close vs CDC-lag incoherence; WAL retention overstated |
| Ops, cost, evolution | 10% | 3.0 | Section 10 is real (shadow mode, migration note); nightly cadence wrong for the domain; no capacity/cost model; no multi-region |
| Communication / artifact | 5% | 3.0 | Complete and organized; coaching tone heavier than ever; no open-questions section; monologue-shaped |

## BLOCKERS (all rated CERTAIN, each confirmed by two reviewers independently)

### 1. The transfer pseudocode violates its own lock-ordering rule; A->B vs B->A deadlocks as written
The board states: "always lock accounts in sorted id order (min(A,B) first) so concurrent A->B and B->A transfers cannot deadlock." The transaction shown directly above it locks sender-first (debit A, then credit B), business order, regardless of id. Interleaving: T1 (A->B) locks A; T2 (B->A) locks B; T1 waits on B; T2 waits on A. Deadlock detection kills one, so it is a retry storm and a p99 killer at peak on guild-pair bidirectional traffic, against the board's own 150ms p99 commitment. Fix: SELECT ... FOR UPDATE both rows in sorted order before the conditional debit, or branch statement order on A < B (credit-before-debit is transactionally fine).

### 2. The REJECTED path destroys its own idempotency anchor
"0 rows -> ROLLBACK, mark REJECTED, 409": the ROLLBACK undoes step 1's transfers INSERT, the idempotency anchor itself. "Mark REJECTED" must then be a second transaction, and in the gap a concurrent retry with the same key inserts fresh, re-executes, and can COMMIT as COMPLETED (funds may have arrived meanwhile) - after which the late "mark REJECTED" either hits a duplicate key or overwrites a COMPLETED row's status while its ledger legs stand. Same key, different outcomes; status/ledger divergence the reconciliation (which checks sums, not statuses) never catches. The elegant part: the ROLLBACK is unnecessary. A 0-row debit changed nothing, so the correct move is UPDATE transfers SET status = REJECTED and COMMIT in the same transaction. The design chose the only sequencing that breaks its own guarantee.

### 3. The advertised invariant cannot detect the failure it advertises catching
Section 11: "Negative balance ever possible? No... The only path to negative is a bug, which reconciliation would surface within a day." Fact-checker's counterexample: a guard-bypass bug lets balance 100 fund two 100-spends. Result: accounts.balance = -100, that account's legs sum to -100, receivers got their credits. Check 1 (global SUM(delta)=0) passes - every transfer's legs still sum to zero. Check 2 (accounts.balance = SUM(entries.delta) per account) passes - both sides read -100. Nothing pages. Detecting it needs a third assertion the board never lists: MIN(accounts.balance) >= 0. Likewise "zero double-spends... provable by invariant" is oversold: SUM(delta)=0 proves conservation, not absence of double-spends.

### 4. Option B (saga) is unrepresentable in the stated data model
The schema admits status COMPLETED | REJECTED only - no in-flight state for the sweeper to inspect. No escrow account and no intent record exist anywhere in the model. "Two legs per transfer" is false under the saga: either 4 legs (via a modeled escrow account, which doesn't exist) or SUM(delta) != 0 mid-flight, which would page the reconciliation continuously on normal in-flight traffic. Sections 2, 5, and 8 cannot all be true at once. Plus the sweeper race: "either completes or releases" can race a delayed receiver-shard credit whose ack was lost - release + landing credit = minted currency; needs terminal-state CAS fencing.

## Significant findings

5. **Credit leg has no existence check.** Step 3's UPDATE has no rows-affected check (only the debit does); transfer to a nonexistent or closed account debits A, commits, and strands money in a ledger leg with no backing account. Account lifecycle (creation, closure with balance, ban) absent from all 12 sections.
6. **Authorization is a comment, not a mechanism.** Body carries "from"; nothing enforces from == actor, so any authenticated client can debit any account as specified. Idempotency keys are globally scoped, so replaying someone else's transfer_id reads their outcome; keys must be scoped (actor, key), and same-key-different-body must 422.
7. **The economy model is P2P-only and the board doesn't know it (staff C1).** A game economy is mostly faucets and sinks - quest rewards, drops, shop purchases. The board's own peak scenario is "(events, drops)", which its data model cannot represent. Double-entry handles mints beautifully via system accounts (and the system account's net position becomes the inflation dashboard) - never introduced, never flagged as a cut. The staff reviewer's opening question kills here: "A quest pays every player 100 gold - what does your design do, and what does SUM(delta)=0 read afterward?"
8. **Integrity detection latency is inverted for the domain (staff C3).** Analytics gets a CDC+Kafka+Flink pipeline for 5-second freshness; the check that protects the economy runs nightly, in a domain where a dupe exploit spreads on Discord and kills the market in hours. The stream for continuous incremental reconciliation is already built and pointed at the wrong consumer. No kill switch, no economy freeze, no velocity caps.
9. **No reversal/clawback story (staff H1).** GM refunds, fraud rollback, bug-dupe cleanup are constant in games; append-only double-entry answers with compensating entries and the board never claims it. COMPLETED | REJECTED is the entire lifecycle.
10. **Multi-region completely absent and unacknowledged (staff C2).** A single-region wallet is defensible for a game; the defense has to be made on the board.
11. **Effectively-once analytics: the one word that makes it true is missing.** Flink crash rewinds the dedupe set with the offsets, so convergence depends on flushes being SET-of-accumulated-total from state, not ADD - "idempotent upserts" is asserted, upsert-of-what never stated. Redis partial hour is worse: "atomic increments" (INCR) is a side effect outside checkpointed state - crash-and-replay double-counts the freshest number on the panel, and the diagram shows no writer feeding Redis at all.
12. **Hour-close has no watermark; recurrence compounds errors forward.** "Hours close once and are immutable" coexists with the board's own 40s CDC lag scenario; a late entry falsifies inflow(h), and closing_balance(h) = closing_balance(h-1) + ... propagates the error through every later point until a rebuild. Their own schema dissolves this: closing_balance(h) = balance_after of the last entry at or before h - self-healing, no recurrence.
13. **Hot sender unsolved.** Sub-accounts summed on read covers the receive-heavy side; the named example (marketplace) is send-heavy, where the no-overdraft guard needs the total across sub-accounts atomically. The hard half (funds rebalancing, or a serialized single-writer with batched debits) unaddressed.
14. **Component count unearned (staff H4).** 1.2K/s average is one well-tuned Postgres primary; the analytics surface (24h flows, immutable hourly buckets) is arguably a one-minute incremental SQL rollup. No "simplest thing and where it breaks" threshold argument anywhere except section 5's genuinely good A/B tradeoff sentence.

## Minor

15. The "24h" flows query spans 24-25h (24 closed buckets + the partial hour); real rolling window needs 23 + partial + aged partial, or say it is an approximation.
16. "Kafka down -> no data loss, only lag" overstates commit-log buffering: binlog purge policies / WAL-filling-the-primary-disk are the real cliff; the claim is only rescued by the rebuild-from-ledger path, which the sentence does not cite.
17. Ledger retention/backup/PITR absent for the source of truth (~20GB/day, multi-TB/yr on OLTP forever); accounts.version declared and never used by any mechanism.
18. Redis restart loses the partial hour with no rebuild-on-miss path; 409 vs 422 for INSUFFICIENT_FUNDS; no amount > 0 or A != B validation; no max-amount/overflow cap.
19. Coaching tone is back and heavier ("Say this first", "Prep drill", "the ops cluster the rubric asks for", "answer it before they ask") and there is no open-questions / what-I'd-cut section: the board is monologue-shaped.

## What the panel credited (convergent)

- The single most important sentence in the problem, correct: the no-overdraft guard fused into the debit statement, "no check-then-act window", with the explicit warning against app-side read-modify-write.
- Double-entry with a checkable conservation invariant and per-account balance = SUM(legs) reconciliation; balance_after as audit gold (concept).
- Idempotency anchored as a unique INSERT inside the money transaction (concept, despite blocker 2's execution).
- CDC off the commit log, correctly justified; Kafka keyed by account_id (per-account ordering claim verified true).
- Fate isolation: "truth path and read models share no fate", "never accept money mutations you cannot commit", replicas-for-display vs primary-for-spending split.
- Section 10 launch plan called genuinely non-checklist by the staff reviewer: shadow mode with batch diff, canary, continuous reconciliation during ramp, and the migration note (snapshot legacy balances as day-zero ledger entries so conservation holds from hour one - "a detail only someone who has done a money migration writes down").
- All arithmetic verified: 1.2K/s, 720 points, ~1.15TB, read ratios. Nothing numeric failed.

## Fix list, in order of leverage

1. Fix the transaction: lock both rows in sorted id order (SELECT FOR UPDATE or branch on id), and make the REJECTED path COMMIT the status update instead of rolling back. Both are 2-line changes to the board's most important code block.
2. Add MIN(balance) >= 0 to reconciliation, and move invariant checks from nightly to continuous (the CDC stream already exists) with a kill-switch/economy-freeze story.
3. Introduce system accounts: mints/sinks as transfers against modeled faucet/sink accounts; conservation invariant becomes the inflation dashboard. This converts the biggest staff-level hole into the design's centerpiece.
4. Add a PENDING status, escrow account, and intent table so Option B is representable; give the sweeper terminal-state CAS fencing.
5. Reversals as compensating entries (never mutation) - one paragraph, huge domain credit.
6. Enforce from == actor; scope idempotency keys to (actor, key); 422 on same-key-different-body.
7. Rows-affected check on the credit leg; account lifecycle paragraph.
8. Replace the hour-close recurrence with closing_balance(h) = balance_after of last entry <= h; state SET-from-state flushes; give Redis a writer and an idempotence story (or drop the Redis hot hour and serve the partial from Flink state).
9. Solve the hot sender or scope it honestly; state the multi-region cut explicitly; add the simplest-design threshold argument; cut the coaching language; add open questions.

## Live probes to practice

1. "Run your pseudocode for concurrent A->B and B->A where A's id sorts after B's. Name each lock, statement by statement. Does it follow your stated rule?"
2. "Your debit matches 0 rows. What exactly is committed when, and what does a concurrent same-key retry observe at each instant of your ROLLBACK-then-mark sequence?"
3. "A quest pays every player 100 gold at event start. Walk me through it, and tell me what SUM(delta)=0 reads afterward." (The staff reviewer's separator: system accounts answer it in thirty seconds; the board as written has no section to recite.)
4. "The rollup crashes 3s after its last checkpoint, having INCR'd Redis and flushed some buckets. What does the 24h flows endpoint return for a touched account after recovery, and what makes each write idempotent?"
5. "Your reconciliation runs nightly. A dupe exploit starts at 2am. Walk me through the next 20 hours of your game economy."