DevEx payouts: the flow, chapter by chapter

WHAT EACH ONE IS

• devex_requests: one row per cashout, the anchor. Its status column is the state machine chapters 1, 2 and 4 move.
• Two states the worker never produces: REJECTED (born terminal at submit when the guard says no) and CANCELLED (the developer cancels while still PENDING; refunds the escrow, guarded so it cannot race a claim; not covered by the chapters below).
• accounts: one row per money holder (dev_777, DEVEX_ESCROW, ROBUX_BURN): id, balance, status (ACTIVE or frozen). Current balances only, no history.
• ledger_legs: the append-only receipts, one row per movement half, pairs sum to zero. The money truth; balances can be rebuilt from it.
• payout_attempts: one row per try, closed once with its outcome, never deleted or re-opened: the audit trail of trying.
• rates (RateVersions on the main page): immutable rate versions. A change is a NEW row, so a locked rate stays true forever.

WHO WRITES WHAT, CHAPTER BY CHAPTER

• Chapter 1 (submit): inserts the devex_requests row, moves accounts, writes two ledger_legs. Reads rates.
• Chapter 2 (claim): updates only devex_requests (status, lease, attempt counter; the reaper later resets status and stamps next_attempt_at).
• Chapter 3 (call): inserts one payout_attempts row per try. Nothing else is written.
• Chapter 4 (settle): the only routine writer that touches four of the five (everything but rates; reconciliation reuses this transaction for a missed settlement): status flip, balances, legs, attempt close, one transaction.
• Chapters 5-6 only read, with ONE exception: reconciliation writes the settlement a processor report proves we missed.
• The WORM archive (the immutable copy, chapter 5) and the DLQ (the parking table of ON_HOLD requests, chapter 4) are derived from these five tables, never sources of truth. The earnings rollups (dashboard totals, chapter 5) are derived the same way from the RobuxLedger, the separate earnings ledger that exists regardless of DevEx.
• Naming: the diagrams write these tables in CamelCase (DevExRequests, LedgerLegs); the text and the SQL use the snake_case table names. Same tables.

WHAT HAPPENS

1. Developer POSTs with an idempotency key. Static validation runs first: malformed body or below the minimum cashout is a 422 before anything external happens.
2. Then the API checks (developer, key): seen before means replay the answer stored the first time (mechanics in Appendix B).
3. Same key with a DIFFERENT body is rejected (409): a key names one exact request, forever.
4. An external eligibility service (KYC, tax forms, fraud state) is asked next, before the transaction below begins. Its full answer is saved (snapshotted).
5. One transaction begins: insert the request row (PENDING).
6. Read the newest row of the rates table. A rate change is a NEW row, never an edit, so this row can be referenced forever. Compute the USD. Store both on the request.
7. Move the Robux: subtract from the developer, add to the escrow holding account. The subtracting SQL itself says "only if enough balance", so the check cannot be raced.
8. Record the move as two bookkeeping rows, called ledger legs: minus on the developer (they are handing over Robux to get dollars), plus on escrow (where it waits). They sum to zero.
9. COMMIT: everything above becomes real at once. The success response already shows the locked rate and the exact dollar amount in cents, and neither can ever change.

WHAT MAKES IT ROBUST

• The fused guard (our name; the standard term is a conditional UPDATE): balance check and debit are one atomic statement. Two racing submissions cannot both pass when the balance covers only one; the second re-checks the debited balance and matches zero rows.
• The anchor (our nickname for the idempotency record: the request row every retry lands on): a guard rejection commits it as REJECTED instead of rolling back, so a retry always finds one recorded outcome.
• Frozen inputs: rate and eligibility are stored on the request. Later rule or rate changes cannot touch this request.
• All or nothing: escrow move, ledger legs, rate, snapshot, status share one COMMIT. No partial submission can exist.

IF WE CRASH RIGHT HERE

• Before COMMIT: nothing exists anywhere. The developer retries cleanly.
• After COMMIT: the Robux sits visibly in escrow and the PENDING row survives. Chapter 2 will find it.

WHAT HAPPENS

1. Workers sweep for PENDING rows.
2. A claim locks the row (FOR UPDATE) while other workers skip locked rows instead of waiting (SKIP LOCKED). One row, one worker, no waiting, no coordinator.
3. The claim also writes a lease ("mine until 12:15" - a timer that outlives a dead worker) and an attempt number (try #1, #2...), then sets PROCESSING.
4. PROCESSING means "a worker owns this row". It has nothing to do with the external payment processor, which is never touched until chapter 3.
5. A reaper job (a cleanup sweep) flips rows with expired leases back to PENDING.

WHAT MAKES IT ROBUST

• FOR UPDATE: two workers can never hold the same request. Not by convention, by the database. SKIP LOCKED: nobody waits for a row it cannot have.
• The lease: a claim is a loan, not ownership. A worker that dies silently just lets its lease expire.
• Attempt numbers: every claim is numbered, so the history shows exactly who tried when.
• Right-sized: at DevEx scale (assume thousands of cashouts/day) one worker suffices. This machinery exists for crash-safety, not throughput.

IF WE CRASH RIGHT HERE

• Worker dies before its claim COMMITs: the claim rolls back, no lease exists, the row is simply still PENDING. Nothing to reap.
• Worker dies after the claim COMMITs: the lease expires, the reaper frees the row, the next worker starts attempt n+1.
• Nothing was sent to the payment processor (the external service that pays out real dollars; chapter 3) yet, so nothing can have been paid.

WHAT HAPPENS

WHAT MAKES IT ROBUST

• One key per request, not per attempt: any retry, from any worker, after any crash, collapses to at most one real disbursement. Per-attempt keys are the classic double-payout bug.
• Definitive-answer rule: a timeout is not an outcome. The row never goes terminal until the processor says something final.
• Attempt row first, call second: a crash between them leaves an honest SENDING row and no side effect. Nothing external happened.
• The reverse order is a bug: call first, record second means a crash can move money with no local record of the attempt. Only reconciliation would ever find it.
• The retry ladder (standard name: a backoff schedule): 15m, 1h, 6h, 1d, 2d. After the last rung the reaper parks the row ON_HOLD instead of PENDING, and a human decides. The product promises resolution inside 5 business days; the whole ladder plus that human-review window fits in that budget.
• Circuit breaker (stop calling a service that keeps failing): workers pause claiming, so unclaimed rows wait as PENDING.
• Rows already claimed just lease-expire back to PENDING. On recovery, workers drain the backlog oldest-first. Unclaimed rows burned nothing; already-claimed rows lost one rung of the ladder.

IF WE CRASH RIGHT HERE

• Crashed after writing SENDING, before calling: nothing external happened. Lease expires; the next attempt adds its own row and calls.
• Crashed after calling, before recording: the SENDING row makes the gap visible (a row with no outcome and an expired lease). The request row is still PROCESSING.
• Next attempt presents the same key. The processor replays its answer. We record it late. Nobody is paid twice.
• Equivalent move: ask the processor's read-only status endpoint for that requestId. Same information, no new payment risk.

WHAT HAPPENS

1. Chapter 3 ends with a definitive outcome in the worker's hand. The same worker, still holding its lease, turns it into money movements.
2. One transaction does four jobs:
   • flip the status column of the devex_requests row: PROCESSING becomes COMPLETED, FAILED, PARTIAL, or ON_HOLD (the ambiguous case below)
   • move the account balances: the developer or the burn account up first, then DEVEX_ESCROW down (the same lock order as submit, Appendix C)
   • write new ledger_legs rows recording that move
   • close the payout_attempts row with the outcome and the processor's reference
3. The status flip is a CAS (compare-and-swap): UPDATE devex_requests SET status='COMPLETED' WHERE id=42 AND status='PROCESSING'.
4. 0 rows updated means the row is no longer ours (someone settled first, or the reaper freed it after our lease expired): stop, touch no money.
5. Where the Robux goes, by outcome:
   • COMPLETED: burn all of it (move it to the burn account, out of the economy forever)
   • FAILED: refund all of it to the developer
   • PARTIAL (paid X of Y): burn robux·X/Y rounded down, refund the remainder
6. An ambiguous answer (their report says PARTIAL but X is missing, negative, or bigger than Y) parks in the DLQ (dead-letter queue). Here that is a parking table, DeadLetterTable, not queue infrastructure: one row per ON_HOLD request, surfaced to humans, for work the system refuses to guess about. It is rebuildable from devex_requests, so it is not a source of truth.
   • the CAS flips the request to ON_HOLD, which takes it out of the claim-and-reaper loop
   • alerts fire, escrow stays put, the attempt row stays open, a human decides before any leg is written

WHAT MAKES IT ROBUST

• Status and money are one commit: no code path can mark a request finished without settling its escrow, or settle without marking.
• CAS on the status: a stale worker or a duplicate settle (the same outcome delivered twice) finds the row already flipped, updates 0 rows, does nothing.
• The escrow invariant: DEVEX_ESCROW balance = SUM of in-flight Robux, checkable at any moment. Every burn matches a settlement line.
• The one-liner: money left the platform if and only if Robux left the economy.

IF WE CRASH RIGHT HERE

• Crash between outcome and settlement: the transaction never committed, the row is still PROCESSING, chapter 3 repeats safely.

WHAT HAPPENS

1. Two databases publish every committed change (CDC = change data capture, tailing the commit log): the RobuxLedger (the earnings ledger: purchase legs, sharded, exists regardless of DevEx) and the PayoutStore (the five tables above).
2. One consumer folds the RobuxLedger stream into the earnings rollups (pre-added totals per developer) that dashboards read.
3. Rollups run seconds stale and carry an asOf timestamp, so stale is visible, never silent.
4. Another streams every PayoutStore change to WORM archive storage (write once, read many): once written, the storage layer refuses edits and deletes, even from admins, until retention expires.

WHAT MAKES IT ROBUST

• Derived, never authoritative: lose the rollups, rebuild from the RobuxLedger; lose the archive copy, rebuild from the PayoutStore.
• The money write path never waits: submissions and payouts do not touch this pipeline; it only reads commits.
• Compliance by stream: the WORM copy is the immutable record regulators ask for, without rebuilding the whole system around an event log (event sourcing).

IF WE CRASH RIGHT HERE

• The pipeline lags or dies: dashboards go stale and say so via asOf. Money paths are untouched.

WHAT HAPPENS

1. Daily, pull the processor settlement report.
2. Diff it against our COMPLETED and PARTIAL rows, in both directions.
3. Their line, no row of ours (money moved that we never recorded): write the missing settlement from their report, then page the on-call at SEV-2 (urgent, next business hour) to find how it got lost.
4. Our COMPLETED or PARTIAL, no line of theirs (we believe we paid; their report shows nothing): page the on-call at SEV-1 (drop-everything severity). Until explained, that is missing money.
5. Continuously, from the CDC stream (chapter 5): re-check the escrow invariant, and that every balance stays at or above zero.

WHAT MAKES IT ROBUST

• External truth check: everything upstream is our own bookkeeping; this is the one chapter that checks it against the outside world, the processor's report.
• Asymmetric severity: money we cannot account for (SEV-1) is worse than money we have not recorded yet (SEV-2). The paging level encodes that judgment.
• Backstop, not front line: chapters 1-4 prevent; this chapter detects whatever survived anyway.

IF WE CRASH RIGHT HERE

• Reconciliation itself fails: nothing corrupts. It is a reader. Rerun it; alert on it being late.

IDIOM 1 · FOR UPDATE, SORTED ORDER: a two-account transfer (teaching example, not part of the payout system)

• Setup: alice (id 1, balance 100) sends bob (id 2) 30. Bob sends alice 10. Same instant.
• Rule: every transaction locks the LOWER id first, no matter who is paying whom.

BEGIN;
SELECT status FROM accounts WHERE id = 1 FOR UPDATE;   -- min(1,2) first, always. the column picked is irrelevant: this SELECT exists only to take the row lock
SELECT status FROM accounts WHERE id = 2 FOR UPDATE;
UPDATE accounts SET balance = balance - 30 WHERE id = 1 AND balance - 30 >= 0;
-- app checks the debit touched 1 row; 0 rows means ROLLBACK and stop (or the credit below mints 30)
UPDATE accounts SET balance = balance + 30 WHERE id = 2;
COMMIT;

• 12:00:00.000 Session 1 locks row 1.
• 12:00:00.001 Session 2 also starts at row 1. Sees the lock. Waits.
• 12:00:00.002 Session 1 locks row 2, updates both, COMMITs.
• 12:00:00.005 Session 2 unblocks and runs its transfer. Nobody deadlocks; the slower one just queues.

THE SAME RACE WITHOUT SORTED ORDER

• 12:00:00.000 S1 (alice→bob) locks its sender: row 1. S2 (bob→alice) locks its sender: row 2.
• 12:00:00.001 S1 wants row 2, waits on S2. S2 wants row 1, waits on S1.
• Each holds what the other needs. Deadlock. Postgres kills one after detection: "deadlock detected".
• Same statements, same locks. The ORDER is the entire fix (standard name: lock ordering).

IDIOM 2 · FOR UPDATE SKIP LOCKED: the payout claim

• Setup: rows 41, 42, 43 are PENDING. Workers A and B run the identical claim at the same instant.

async def claim_next(pool):
    async with pool.connection() as conn:      # borrowed from a connection pool, not opened fresh
        async with conn.transaction():         # framework does BEGIN / COMMIT / ROLLBACK

            # 1. pick ONE free row and lock it: skip busy rows, never wait
            row = await conn.fetchrow(
                "SELECT id, dev, robux, usd_minor FROM devex_requests "
                "WHERE status = 'PENDING' AND next_attempt_at <= now() "  # the retry-ladder wait, enforced
                "ORDER BY created_at "                      # created_at: stamped by column default in chapter 1
                "LIMIT 1 FOR UPDATE SKIP LOCKED")
            if row is None:                    # every PENDING row is locked or gone
                return None                    # caller sleeps a beat, polls again

            # 2. claim the row we are holding: still locked, nobody can race us
            await conn.execute(
                "UPDATE devex_requests "
                "SET status = 'PROCESSING', "
                "    lease_until = now() + interval '15 minutes', "
                "    attempt = attempt + 1 "
                "WHERE id = $1", row.id)      # $1 = the id the SELECT above locked (say 42)

    # leaving the block ran COMMIT: lock released, claim durable
    return row                                 # caller sends row 42 to the processor

• 12:00:00.000 A reaches row 41 first, locks it. A gets 41.
• 12:00:00.000 B reaches row 41, sees the lock, does NOT wait: skips to 42, locks it. B gets 42.
• 12:00:00.002 Both COMMIT their claims. Row 43 waits for the next sweep.
• 12:07:00 Worker A crashes. Nothing happens yet.
• 12:15:00 A's lease expires. The reaper flips row 41 back to PENDING. Next claim is attempt 2 (claimable from 12:30: the ladder's first rung, chapter 3).

THE SAME CLAIM WITHOUT SKIP LOCKED

• B queues behind A for row 41 (a row it can never have) instead of taking 42.
• Every worker serializes behind the first one. The pool becomes single-file.

ONE PRIMITIVE, TWO IDIOMS

• Money transfers: lock everything you touch, in one global order. Wait politely.
• Work queues: lock one free thing, skip the busy ones. Never wait.
• When is SELECT FOR UPDATE needed at all? Only when there is a gap between reading and writing that the lock must cover.
• A plain UPDATE takes the SAME row lock by itself and holds it to COMMIT. That is why the account updates in Appendices B and C need no SELECT: read, guard, and write are one statement, no gap.
• Idiom 2's gap: the worker must read WHICH row to take before it can write to it. The SELECT is the picking; FOR UPDATE makes the pick a claim.
• Lock ORDER matters either way: consecutive UPDATEs also take locks one at a time and hold them, so opposite orders can still deadlock.
• Name both idioms in an interview and row locking is covered as a topic.

THE HANDLER, PRODUCTION SHAPE (Python-ish)

async def submit_devex(dev_id, idem_key, robux):
    if robux < MIN_CASHOUT:                    # static validation first: nothing external, nothing written
        return 422, {"error": "BELOW_MINIMUM"}
    # 0. replay check FIRST: a retry must get the stored answer even if
    #    eligibility has changed since the original request went through
    async with pool.connection() as conn:      # same pool as step 2: under sharding this call is routed too
        prior = await conn.fetchrow("SELECT body_hash, http_status, stored_response "
                                   "FROM devex_requests WHERE dev=$1 AND idem_key=$2", dev_id, idem_key)
    if prior:
        if prior.body_hash != digest(robux):
            return 409, {"error": "KEY_REUSED"}
        return prior.http_status, prior.stored_response

    # 1. external call OUTSIDE the transaction: own timeout, own circuit breaker
    elig = await eligibility.check(dev_id, timeout=2.0)
    if not elig.ok:
        return 422, {"error": "NOT_ELIGIBLE", "reasons": elig.reasons}

    async with pool.connection() as conn:      # borrowed from a pool, not opened fresh
        async with conn.transaction():         # framework does BEGIN / COMMIT / ROLLBACK

            # 2. the anchor, race-safe: insert-or-detect in one statement
            req = await conn.fetchrow(
                "INSERT INTO devex_requests (dev, idem_key, robux, body_hash, status) "
                "VALUES ($1,$2,$3,$4,'PENDING') "
                "ON CONFLICT (dev, idem_key) DO NOTHING RETURNING id",
                dev_id, idem_key, robux, digest(robux))   # digest: a stable hash of the canonical body (sha256), never Python's hash()
            # column defaults fill the rest: attempt 0, next_attempt_at now(), created_at now()
            if req is None:                    # two racing FIRSTs on one key: the loser lands here
                prior = await conn.fetchrow("SELECT * FROM devex_requests "
                                            "WHERE dev=$1 AND idem_key=$2", dev_id, idem_key)
                if prior.body_hash != digest(robux):
                    return 409, {"error": "KEY_REUSED"}
                return prior.http_status, prior.stored_response    # the replay

            # 3. rate lock + USD, still inside the same transaction
            rate = await conn.fetchrow("SELECT * FROM rates WHERE valid_from <= now() "
                                       "ORDER BY valid_from DESC LIMIT 1")
            usd_minor = floor(robux * rate.minor_per_robux)  # cents per Robux; floor: exact cents, never overpay

            # 4. the fused guard: check + debit, one statement
            n = await conn.execute(
                "UPDATE accounts SET balance = balance - $2 "
                "WHERE id = $1 AND status='ACTIVE' AND balance - $2 >= 0", dev_id, robux)
            if n == 0:
                # zero rows means frozen account OR insufficient balance. production re-reads the
                # account once to say WHICH; this sketch records the common case only
                # business rejection: RETURN, do not RAISE - raising would roll back
                # the anchor, and the anchor must survive for retries
                resp = {"error": "INSUFFICIENT_BALANCE"}
                await conn.execute("UPDATE devex_requests SET status='REJECTED', "
                                   "http_status=422, stored_response=$2 WHERE id=$1", req.id, resp)
                return 422, resp   # the retry replays exactly this, via step 0

            await conn.execute("UPDATE accounts SET balance = balance + $1 "
                               "WHERE id = 'DEVEX_ESCROW'", robux)   # the credit: no guard needed
            await conn.execute("INSERT INTO ledger_legs ... ")     # the receipts: -dev, +ESCROW
            resp_body = {"requestId": req.id, "status": "PENDING", "rateId": rate.id, "usdMinor": usd_minor}
            await conn.execute("UPDATE devex_requests SET rate_id=$1, usd_minor=$2, "
                               "elig_snapshot=$3, http_status=201, stored_response=$5 "
                               "WHERE id=$4", rate.id, usd_minor, elig.json, req.id, resp_body)
            # http_status + stored_response: what a future retry replays
    # leaving the block runs COMMIT (clean exit) or ROLLBACK (exception).
    # by this line the commit has already happened.
    return 201, resp_body   # the exact object a retry will replay

THE REAL-WORLD DETAILS THE WHITEBOARD HIDES

WHAT BREAKS IN REAL DEPLOYMENTS, AND WHERE THIS DESIGN CATCHES IT

• Deploy rolls mid-request: transaction never committed, nothing exists, client retries cleanly.
• Eligibility service is slow: only the pre-transaction call waits; no database locks are held during it.
• Two app servers race the same key: ON CONFLICT makes one the writer; the other reads and replays.
• Clock skew between servers: irrelevant here - rate selection uses the database's now(), one clock.

ONE EXAMPLE, CARRIED THROUGH EVERY CASE

• Request 42: developer dev_777 cashed out 1000 Robux at the locked rate, worth 350 usdMinor ($3.50). One id, three spellings: requestId in the API = id on devex_requests = request_id on the child tables.
• Chapter 1 already moved the Robux into escrow. Chapter 4 decides where it goes from there.

THE POSITION SETTLEMENT INHERITS (after chapters 1-3)

devex_requests
  id=42  dev=dev_777  robux=1000  usd_minor=350  status='PROCESSING'

accounts
  dev_777        4000    # had 5000; chapter 1 debited 1000
  DEVEX_ESCROW   1000    # the in-flight 1000 waits here
  ROBUX_BURN        0    # the exit door of the economy

ledger_legs (request 42 so far: chapter 1 wrote this pair)
  leg=901  account=dev_777       amount=-1000  reason='SUBMIT'  request=42
  leg=902  account=DEVEX_ESCROW  amount=+1000  reason='SUBMIT'  request=42

payout_attempts
  id=7  request=42  attempt=1  processor_key='42'  state='SENDING'
  # the key IS the requestId (chapter 3's rule). still SENDING: the definitive answer
  # is in the worker's hand; chapter 4's transaction writes it while settling

CASE: PROCESSOR SAYS FAILED: full refund

• Meaning: no dollars left the platform, so all 1000 Robux must go back.
• One transaction, four jobs (six statements):

async def settle_failed():                      # same frame as Appendix B: the with block IS the transaction
    async with pool.connection() as conn:
        async with conn.transaction():         # framework does BEGIN / COMMIT / ROLLBACK

            # 1. the CAS: claim the right to settle, or learn someone already did
            n = await conn.execute(
                "UPDATE devex_requests SET status='FAILED' "
                "WHERE id=42 AND status='PROCESSING'")
            if n == 0:                        # someone settled first
                return                        # stop, touch no money

            # 2. move the Robux back (no overdraft guard needed: escrow always covers, by invariant).
            #    developer row first, then escrow: the SAME lock order as chapter 1's submit,
            #    per idiom 1's principle (ONE global order; here a fixed role order, not lower-id first):
            #    opposite orders between submit and refund could deadlock
            await conn.execute("UPDATE accounts SET balance = balance + 1000 WHERE id='dev_777'")
            await conn.execute("UPDATE accounts SET balance = balance - 1000 WHERE id='DEVEX_ESCROW'")

            # 3. the receipts: two NEW legs. chapter 1's legs are never edited or deleted.
            await conn.execute("INSERT INTO ledger_legs VALUES (903, 'DEVEX_ESCROW', -1000, 'REFUND', 42)")
            await conn.execute("INSERT INTO ledger_legs VALUES (904, 'dev_777',      +1000, 'REFUND', 42)")

            # 4. close the attempt with the processor's reason
            await conn.execute("UPDATE payout_attempts SET state='FAILED', error='account_closed', processor_ref='pp_9f31' WHERE id=7")

    # leaving the block ran COMMIT: all six statements land, or none do

after COMMIT:
  dev_777        5000    # whole again, as if the request never happened
  DEVEX_ESCROW      0    # this request's slice is exactly zero
  legs for request 42: -1000 +1000 -1000 +1000  # sum = 0, story preserved

CASE: PROCESSOR SAYS PARTIAL: paid 210 of 350 usdMinor

• Meaning: 60% of the dollars went out, so 60% of the Robux must leave the economy.
• Burned share: floor(1000 · 210/350) = 600. Refund the remaining 400.
• Floor rounding sends the fractional Robux back to the developer, never into the burn.
• Same shape: CAS to 'PARTIAL', then TWO pairs of legs instead of one:

# alternative outcome for the SAME request 42 (instead of the FAILED case, not after it),
# so leg numbering starts at 903 again
async def settle_partial():                     # paid 210 of 350: burn 600, refund 400
    async with pool.connection() as conn:
        async with conn.transaction():

            # 1. the CAS: claim the right to settle, or learn someone already did
            n = await conn.execute(
                "UPDATE devex_requests SET status='PARTIAL' "
                "WHERE id=42 AND status='PROCESSING'")
            if n == 0:
                return                        # someone settled first: stop, touch no money

            # 2. balances: developer, then escrow, then burn. "global" = EVERY transaction in the
            #    codebase uses this same sequence (skipping rows is fine, reordering is not),
            #    so no cycle of waiters can ever form (idiom 1's rule, with names)
            await conn.execute("UPDATE accounts SET balance = balance + 400  WHERE id='dev_777'")
            await conn.execute("UPDATE accounts SET balance = balance - 1000 WHERE id='DEVEX_ESCROW'")
            await conn.execute("UPDATE accounts SET balance = balance + 600  WHERE id='ROBUX_BURN'")

            # 3. the receipts: TWO pairs this time, each summing to zero
            await conn.execute("INSERT INTO ledger_legs VALUES (903, 'DEVEX_ESCROW', -600, 'BURN',   42)")
            await conn.execute("INSERT INTO ledger_legs VALUES (904, 'ROBUX_BURN',   +600, 'BURN',   42)")
            await conn.execute("INSERT INTO ledger_legs VALUES (905, 'DEVEX_ESCROW', -400, 'REFUND', 42)")
            await conn.execute("INSERT INTO ledger_legs VALUES (906, 'dev_777',      +400, 'REFUND', 42)")

            # 4. close the attempt with what was actually paid
            await conn.execute("UPDATE payout_attempts SET state='PARTIAL', paid_minor=210, processor_ref='pp_9f32' WHERE id=7")

    # leaving the block ran COMMIT: all nine statements land, or none do

after COMMIT:
  dev_777        4400    # got 400 back
  DEVEX_ESCROW      0    # emptied for this request, again
  ROBUX_BURN      600    # 600 Robux left the economy, matching the $2.10 that left the platform

CASE: PROCESSOR SAYS COMPLETED: for symmetry

• CAS to 'COMPLETED', one pair of legs: DEVEX_ESCROW -1000, ROBUX_BURN +1000.
• The one-liner: dollars left the platform, so exactly that many Robux left the economy.

RULES THAT HOLD IN EVERY CASE

• Append-only ledger: a refund is two new rows, never an edit of chapter 1's rows. The history stays readable.
• Escrow drains to exactly zero for the request, whichever case fires. That is the per-request escrow invariant.
• Legs come in pairs summing to zero, and the request's whole set sums to zero: nothing minted, nothing lost.
• Lose the CAS, write nothing: status flip, account updates, legs, and the attempt close share one transaction.
• Ambiguous writes NO legs: the CAS parks the request as ON_HOLD and the 1000 sits in escrow until a human picks one of the three cases above. The human path runs the same transaction with AND status='ON_HOLD' in its CAS.

THE QUESTION, TAKEN AT FACE VALUE

• The interviewer says: your submission API runs at 10K QPS and jumps to 100K. What breaks first?
• One honest sentence first: real DevEx volume is thousands/day, so this is a hypothetical. Then answer it straight: dodging the premise scores zero.
• The drill is about WHERE you look: the design's own global rows, its one box, its synchronous external calls (eligibility, then the processor), in that order.

FIRST BREAK: THE GLOBAL ESCROW ROW

• Chapter 1's transaction credits DEVEX_ESCROW, ONE row, on every submission, from every developer.
• A row update holds that row's lock through COMMIT. One row serializes at low thousands/sec: dead before the 10K baseline, let alone 100K.
• Symptom chain: lock queue → latency spike → connection pile-up → pool exhaustion → everything degrades, including reads.
• The fix: escrow PER DEVELOPER: escrow(dev_777), one row each. Nothing about escrow ever needed to be one row; the balance was always an aggregate of legs.
• The invariant gets BETTER: escrow(dev) = SUM of that developer's in-flight Robux, checkable per developer, no global read.
• Same medicine at settlement: ROBUX_BURN becomes a sharded counter: one sub-account per shard, credited locally; reconciliation sums the sub-accounts daily. (Avoid calling it striping in a payments room, where Stripe is a company.)
• Why different keys: escrow splits per DEVELOPER because the invariant is checked per developer. Burn splits per SHARD because it is only ever read as a sum. The split key follows how the number is read.

SECOND BREAK: ONE POSTGRES, AND WHY SHARDING IS PAINLESS HERE

• 100K txns/sec of multi-row writes is beyond one box: shard by developerId.
• THE point to say out loud: a payout never crosses developers. Request row, developer account, developer escrow, both legs: all belong to ONE developer.
• So every submission transaction lands whole on one shard: the one-transaction write survives sharding untouched. No 2PC (two-phase commit, a cross-database commit protocol), no saga (a chain of local transactions with compensations), no relaxed invariant.
• Contrast in one sentence: flows that touch two parties (transfers between users) lose this property and need cross-shard machinery. Ours does not. Knowing WHICH kind you have is the senior signal.
• What stays global: rates (read-only, cached everywhere, no lock) and the reconciliation report (already a daily batch).
• What this looks like in real life: each shard is a separate Postgres server with IDENTICAL tables: every shard has its own devex_requests, same name, same DDL, different rows.
• The app holds the routing map: hash(developerId) % 64 buckets → connection string. Sharding picks the SERVER before any SQL runs; the SQL itself never changes.
• Buckets outnumber servers (64 buckets, 4 boxes) so growing means moving buckets and flipping the map, not re-hashing rows. Migrations run once per shard.

SHARD_POOLS = {0: pool_a, 1: pool_b, ...}      # one connection pool per Postgres server
ROUTING = {bucket: shard_id}                       # 64 buckets → N servers, hot-reloadable config

def pool_for(dev_id):
    bucket = crc32(dev_id.encode()) % 64   # a STABLE hash. never Python's hash(): it is salted per process
    return SHARD_POOLS[ROUTING[bucket]]

async def submit_devex(dev_id, idem_key, robux):
    async with pool_for(dev_id).connection() as conn:   # THE targeting: which server we dial
        async with conn.transaction():
            # ... every statement from Appendix B, byte-identical ...
            # the SQL never knows which shard it is on

• One changed name versus the unsharded handler: pool became pool_for(dev_id), at both call sites (step 0 and step 2). That is the entire blast radius, and it is why this sharding is called painless.
• Is app-side routing normal? Yes: it is THE standard for plain Postgres/MySQL (Instagram, Notion ran exactly this). Plain Postgres has no native sharding, so your code owns the hash.
• The other two homes for the hash: a middleware owns it (Vitess, Citus: the app dials ONE endpoint, the proxy routes by shard key), or the store owns it (DynamoDB, Cassandra, Spanner: the partition key in each request IS the routing; you never see shards).
• Unifier: something always hashes a key to pick a destination. The design choice is WHICH LAYER owns that hash: your code, a proxy, or the database.

ADDING AND REMOVING SHARDS: THE BUCKET-MOVE PLAYBOOK

• Shards are logical from day one: 64 buckets exist when there is ONE server (all route to it). A new shard is an empty Postgres with the same migrations, ready to receive buckets.
• Add a node: provision + migrate, pick buckets to move (say 3-4 from each of the four boxes, 13 in total), backfill each via snapshot + logical replication catch-up.
• Cut over PER BUCKET: pause that bucket's writes, drain, flip ROUTING, resume. Seconds per bucket.
• The domain gift: our SLA is 5 business days. Pausing 1/64th of submissions for 30 seconds is invisible. The payout system's slowness makes its resharding easy; a feed or counter cannot afford this, we can.
• Verify before deleting: row counts AND per-developer escrow sums must match on both sides. The invariant doubles as a migration checksum.
• Removing a node is the same playbook reversed: move its buckets out, verify, decommission.
• Manual? The steps are scripts; the trigger is a human with a runbook. That is the true cost of code-owned hashing: Vitess/Citus sell this playbook as a workflow, Cockroach/Spanner run it continuously without asking.

THIRD BREAK: THE SYNCHRONOUS ELIGIBILITY CALL

• Chapter 1 calls the eligibility service once per submission, before the transaction. At 100K QPS that is 100K outbound calls/sec into someone else's service.
• Fix without breaking frozen-at-submit: cache the verdict per developer with a short TTL, invalidated by eligibility-change events. You still snapshot WHATEVER verdict you used onto the request; the freeze rule is about recording, not about who answered from where.
• Degraded mode stays honest: eligibility service down means submissions queue or 503, never "skip the check".

FOURTH BREAK: THE PROCESSOR ITSELF

• No payment processor disburses 100K/sec, and their rate limits arrive long before that.
• The move is batching: one processor call carries many requests. Idempotency adapts: the batch gets a batchId key, each line keeps its requestId, so replay dedupes at both levels.
• Settlement then fans a batch answer back out to per-request CAS + legs, unchanged.

THE MAP AFTERWARD

• Sharded by developer: devex_requests, payout_attempts, developer accounts, escrow(dev), ledger legs. Each shard is a complete miniature of chapters 1-4's design.
• Sharded counter: ROBUX_BURN, one sub-account per shard, summed by reconciliation.
• Global but read-only: rates, cached.
• Unchanged: every per-request mechanism: the anchor, the guard, SENDING-before-call, the CAS, the ladder. Scale changed the layout, not the rules.

THE CLOSE

• "First the global rows die, so make escrow per-developer. Then the box dies, so shard by the transaction's natural boundary, the developer, and because a payout never crosses developers, every guarantee survives. The externals break last: caching handles the eligibility call, batching handles the processor."