← all topics DevEx payouts board · say it in this order 1 Problem2 Entities + API 3 High-level design4 Deep dives 5 Final + levelsreset
SAY THIS SENTENCE FIRST

"Robux is a closed double-entry economy, and DevEx is the one door out of it. A request escrows Robux atomically at submission, and what happens to that escrow, destroyed (burned) or refunded, is decided by what real dollars actually did."

STEP 1 OF 5

Understanding the problem

4 min
💸 What is DevEx?
  • Creators earn Robux when players spend in their games.
  • DevEx converts earned Robux into real US dollars via an external payment processor.
  • It is the only place the closed in-game economy touches real money.

Functional Requirements

  1. Developers should be able to see Robux earnings per game in near real time.
  2. Developers should be able to submit a DevEx request (a minimum cashout amount and eligibility rules apply).
  3. The system should convert at the rate in effect at submission, not at processing.
  4. Developers should be able to track request status and full history.

Below the line (out of scope):

  • The eligibility rule engine itself. We consume it as a service.
  • Tax documentation and reporting.
  • Multi-currency rails; the processor internals.

Non-Functional Requirements

Ask about scale first: it splits the problem in half
  • Earnings: tens of millions of purchases/day × 3 legs each: hundreds to ~1k ledger writes/sec. Spiky.
  • Payouts: thousands/day. Under 0.1/sec. Tiny but high-stakes.
  • You are designing two systems with opposite profiles. Say that.
  1. The system should never allow over-withdrawal. Balance at submission: strongly consistent.
  2. The system should never lose or mutate a financial record. Every decision explainable years later.
  3. The system should finish payouts within 5 business days, despite processor downtime, rejections, and partial payments (paid X of a requested Y).
  4. The system should serve earnings reads cheaply. Eventually consistent display is fine, labeled with a freshness timestamp (asOf).
DONE WHEN: interviewer nods at the scale split: earnings are high-volume, payouts are tiny but high-stakes. That split shapes everything.
STEP 2 OF 5

Entities + API

4 min

Defining the Core Entities

  • Developer Account: holds the withdrawable Robux balance and a status (ACTIVE / FROZEN).
  • Earnings Ledger Entry: one leg (one half of a transfer, + or -) of a Robux movement. Double-entry. The truth.
  • DevEx Request: one cash-out of Robux to USD, carrying a locked rate, an eligibility snapshot and a status.
  • Exchange Rate Version: immutable (rate, valid-from) row. Requests point at one.
  • Payout Attempt: one processor call and its outcome. Immutable.
  • System Accounts: DEVEX_ESCROW holds in-flight Robux. ROBUX_BURN destroys it when a payout completes.
💡 The sentence to plant early

"Robux is a closed double-entry economy, and DevEx is the one door out. A request escrows Robux atomically at submission. Burn or refund is decided by what real dollars actually did."

API or System Interface

POST /v1/devex-requests          Idempotency-Key: <key>   // unique per (developer + key) pair
{ "robux": 100000 }
→ 201 { requestId, PENDING, rateId, usdMinor }   // usdMinor = cents; rate + USD locked NOW
→ 422, body: { error: INSUFFICIENT_BALANCE | BELOW_MINIMUM | NOT_ELIGIBLE, reasons }
→ 409, body: { error: KEY_REUSED }   // same key, different body. same key + same body: replay the stored 201

GET    /v1/devex-requests?cursor=            // history
GET    /v1/devex-requests/:id                // status + every attempt
DELETE /v1/devex-requests/:id                // cancel; PENDING only; 409 after
GET    /v1/earnings?game=&granularity=day    // display; eventual; carries asOf
  • The 201 already carries the locked rate and computed USD.
  • Both are fixed inside the submission transaction. The amount can never drift after.
DONE WHEN: you have said the opening sentence again, pointing at ESCROW and BURN.
STEP 3 OF 5

High-level design

10 min, end to end, no dives yet

1) Developers should be able to see earnings in near real time

Earnings in near real time: purchases write ledger legs, the dashboard sums them A player's purchase writes three legs to RobuxLedger, and the same legs sum into DeveloperAccount.balance, the exact withdrawable amount. The developer asks EarningsAPI for earnings; EarningsAPI sums the developer's legs per game on RobuxLedger and returns the totals with an asOf stamp. Player RobuxLedger (truth) one purchase = 3 legs-400 Player+280 DeveloperShare+120 PlatformCutlegs sum to zero: double-entryhundreds to ~1k writes/sec EarningsAPI sums legs per game Developer DeveloperAccount.balance the exact withdrawable Robux 1. buys item 400 Robux in game G 2. GET /earnings 3. sums legs per game,on every read 4. totals + asOf legs sum into the balance the not-so-happy path: every dashboard read sums legs on the ledger that takes hundreds to ~1k purchase writes/sec.the scale-asymmetry deep dive takes those reads off the ledger.
  • Every purchase writes double-entry legs: minus player, plus developer share, plus platform cut.
  • Legs sum to zero, so the economy conserves by construction.
  • The earnings read sums the developer's legs per game on the ledger and returns the totals with asOf.
  • The same legs sum into the developer's balance, the exact withdrawable Robux.
  • Every dashboard read now competes with hundreds to ~1k purchase writes/sec on the ledger; the scale-asymmetry dive takes those reads off it.

2) Developers should be able to submit a DevEx request

Submitting a DevEx request: replay check, eligibility once, then one commit The developer posts to DevExAPI with an Idempotency-Key. DevExAPI checks for a replay on DevExRequests, asks EligibilityService once, then commits one transaction: insert the PENDING request, lock the rate and USD from RateVersions, check the account is ACTIVE and covers the amount, and debit the developer into escrow. The developer gets 201 with the rate and USD locked. Developer DevExAPI static checks at the door:BELOW_MINIMUM answered here EligibilityService (external) rules + identity, snapshotted ONE TRANSACTION - single COMMIT everything the money depends on is fixed here a. DevExRequests insert PENDING: the idempotency recordunique index on (developer, key) b. RateVersions (immutable) read current rate; store rateId + usdMinor c. Accounts + Escrow check ACTIVE and balance >= robux,then debit developer, credit DEVEX_ESCROW 1. POST /v1/devex-requests Idempotency-Key + {robux:100000} 5. 201 rate+USD locked NOW requestId, PENDING,rateId, usdMinor 2. replay check first same key+body: stored 201new body: 409 KEY_REUSED 4. then ONE COMMIT one txn: a, b, c below 3. eligibility ONCE not eligible: 422 + reasons the not-so-happy path: account frozen or balance short -> mark REJECTED with its reason, commit, 422. nothing was debited,nothing to refund; the idempotency record survives for retries.two submissions racing for one balance can both pass the check: the over-withdraw deep dive closes that.
  • Replay check first: (developer, key) seen with matching body hash → return the stored answer, even if eligibility has since changed. Body differs → 409.
  • Then the eligibility check (external call). Full result snapshotted.
  • Then ONE transaction does the money work: insert the request, store the rate and USD on it, check the balance, debit the developer into ESCROW. ESCROW is the system account holding in-flight Robux.
  • Two submissions racing for one balance can both pass the check; the over-withdraw dive closes the gap.

The transaction, step by step:

  1. Insert the request row. It is the idempotency record, the row a retry lands on.
  2. Read the current rate version. Compute the USD. Store both on the request.
  3. Check the account is ACTIVE and its balance covers the amount, then debit developer, credit ESCROW.

If the check fails (the branch of step 3):

  • Frozen account or insufficient balance: one error code each.
  • Mark REJECTED, commit, 422. Nothing was debited, so there is nothing to refund.
  • The request row survives as the idempotency record, so a retry finds it (unique index on developer + key).
TRIGGER: "which rate applies" / "rules changed after they submitted"

3) The rate is the one at submission

RateVersions: a change is a NEW immutable row, never an UPDATE Three immutable rate rows sit above a timeline: v41 from Mar 1, v42 from Jun 1, v43 from Jun 12. A DevExRequest submitted on Jun 10 stores a foreign key to v42 and its USD amount. PayoutWorker reads only that request row on Jun 14, so v43 landing mid-flight changes nothing. RateVersions: a change is a NEW immutable row, never an UPDATE v41 0.0030 USD/Robuxvalid-from Mar 1 v42 0.0035 USD/Robuxvalid-from Jun 1 v43 0.0038 USD/Robuxvalid-from Jun 12lands mid-flight, still ignored time Mar 1 Jun 1 Jun 10 Jun 12 Jun 14 submission processing DevExRequest robux = 100000rateId = v42   usdMinor = 35000   written inside the submission txn PayoutWorker runs days later 1. FK: rateId points at ONE immutable row 2. days later: reads the request sees rateId + usdMinor only the guarantee is a FOREIGN KEY, not a convention the not-so-happy path: a rate change lands mid-flight? irrelevant.processing never consults the rate table, only the request row.
  • A rate change is a NEW immutable row. Never an update.
  • The request stores which rate row it used and the computed USD.
  • Processing days later reads the request. It cannot see any other rate.
  • The guarantee is a foreign key, not a convention.

4) Developers should be able to track status and history

DevExRequest state machine A request starts PENDING. A worker picks it up into PROCESSING, or a cancel moves it to CANCELLED. From PROCESSING it ends COMPLETED, PARTIAL, FAILED or ON_HOLD. Every processor call appends one row to PayoutAttempts, and GET /devex-requests/:id returns that table. REJECTED is born terminal at submit. DevExRequest state machine PENDING PROCESSING PayoutAttempts (immutable) n | outcome 1 | timeout 2 | COMPLETED one immutable row per processor call GET /devex-requests/:id CANCELLED COMPLETED PARTIAL FAILED ON_HOLD worker picks it up calls the processor cancel (PENDING only) refund the escrow each call appends ONE row reads the table verbatim audit trail and UI at once paid in full burn the escrow paid X of Y split, see theprocessor-failure dive definitive no refund the escrow unclear answer a human decides REJECTED born terminal at submit: the balance check said no.nothing was escrowed, nothing to refund. it never enters this machine. the not-so-happy path: a worker that crashes after calling the processor, or a status change written without its escrow legs,can pay twice or strand Robux: the processor-failure deep dive handles both.
  • State machine: PENDING → PROCESSING → COMPLETED | PARTIAL | FAILED | ON_HOLD. Plus PENDING → CANCELLED.
  • REJECTED is born terminal at submit (the balance check said no; nothing escrowed).
  • A payout worker picks up a PENDING request, sets PROCESSING, calls the processor, and sets the ending status from its answer.
  • A COMPLETED request burns its escrowed Robux; FAILED and CANCELLED refund it.
  • Every processor call writes one immutable attempt row.
  • The history endpoint returns that table verbatim. Audit trail and UI at once.
  • A crash between calling the processor and recording its answer, or a status change written without its escrow legs, can pay twice or strand Robux; the processor-failure dive handles both.
DONE WHEN: the interviewer picks a box to open. Let them steer from here.
STEP 4 OF 5

Deep dives

~20 min, interviewer steers
TRIGGER: "two submissions race for the same balance" / "walk me through submit"

1) How do we guarantee a developer can never over-withdraw?

  • The race: two submissions, one 100K balance, 80K each. Both pass app-level checks.
Bad Solution: check-then-act in the application
  • Read balance, compare in code, then debit.
  • Between read and write, the other request commits.
  • Both pass. You paid 160K from 100K. Interviewers plant this deliberately.
Good Solution: serializable transactions
  • Run everything at the strongest isolation level, retry on conflict.
  • Correct, but you now handle retry storms in the app.
  • And you pay the maximum price for a path needing exactly one guarantee.
Great Solution: a conditional debit (the check inside the UPDATE)
UPDATE accounts SET balance = balance - :robux
WHERE id = :dev AND status = 'ACTIVE'
  AND balance - :robux >= 0
  • The balance check lives inside the debit UPDATE, so check and mutation are ONE atomic statement under the row lock.
  • Of two concurrent submissions, the second waits, re-checks the already-debited balance, matches zero rows, and is rejected.
  • No check-then-act window exists.
  • "Strongly consistent balance" here means one line of SQL, not a distributed protocol.
  • The minimum-cashout check is NOT here: it is static validation at the API edge (a constant cannot be raced). The WHERE clause holds only the racy checks.
  • When the debit matches zero rows, re-read the account once to say WHY: frozen account or insufficient balance, one error code each. Then mark REJECTED, commit, 422.
TRIGGER: "should this be event sourced" / "what do you watch"

2) How is every payout decision auditable years later?

Bad Solution: mutable rows plus application logs
  • Update the request in place; log what happened.
  • Logs rotate. Updates destroy history.
  • "Why did this pay $3,500 in 2024?" has no answer a regulator accepts.
Good Solution: event-source everything
  • Rebuild ALL state only by replaying an event log.
  • Fully auditable, but you adopted a whole architecture: snapshots, upcasting (migrating old stored events to new schemas), replay tooling.
  • This domain can answer the question more cheaply.
Great Solution: event-sourced where it matters, immutable inputs everywhere
  • Event-source where it matters, the money, and archive the change stream everywhere else.
  • The ledger legs + attempts table already ARE an append-only event log.
  • Rates are immutable versions, and eligibility is snapshotted onto the request.
  • Request status is a normal column, and every change streams via CDC to a write-once, read-many (WORM) archive.
TRIGGER: "the processor is slow/down" / "your worker crashes mid-payout" / "the processor paid half" / "what ends a request" / "processor is down for 6 hours" / "their report disagrees with you"

3) How do we finish in 5 business days when the processor fails, rejects, or half-pays?

Bad Solution: retry each failed call with a new key
  • The worker calls the processor, and on a timeout retries with a fresh idempotency key per attempt.
  • A timeout does not mean the payment failed. If the first call paid, the retry pays again.
  • Marking the timed-out request FAILED and refunding is the same bug in reverse: the developer may be paid and refunded.
Good Solution: one key per request, the requestId, on every retry
  • Every retry sends the requestId as the processor's idempotency key, so retries collapse to at most one disbursement.
  • Still wrong:
    • A worker that crashes after the call leaves the row PROCESSING with nobody to finish it.
    • A status change written apart from its escrow legs can strand the Robux or settle them twice.
    • Nothing catches a payment the retries missed.
Great Solution: leased claims, one processor key, escrow settled with the status, daily reconciliation
  • Every retry sends the processor the same idempotency key, the requestId, so retries collapse to at most one disbursement.
  • An ambiguous call (a timeout, or a crash before recording the answer) stays PROCESSING until the processor gives a definitive answer, with retries at 15m, 1h, 6h, 1d, 2d inside the 5 business days.
  • Each terminal state settles the escrow (the Robux held since submission) in the same transaction as the status change: COMPLETED burns it, FAILED refunds it, PARTIAL splits it.
  • Daily reconciliation against the processor's settlement report, in both directions, catches what retries missed.
  • Volume is thousands/day, so ONE worker is enough: leases, one processor key and reconciliation exist for crash-safety, not throughput. Saying that is the judgment signal.
  • Claiming work:
    • Workers sweep PENDING with FOR UPDATE SKIP LOCKED. The claim takes a lease and an attempt number and sets PROCESSING.
    • A reaper flips expired-lease rows back to PENDING, the one backward transition. A definitive answer is the only way forward.
    • "ONE worker" means one logical consumer; run a spare; SKIP LOCKED makes extras safe.
  • Calling the processor:
    • The idempotency key IS the requestId. A SECOND key, one hop out: the client's Idempotency-Key dedupes developer→us; this one dedupes us→processor. Every retry collapses to at most one disbursement.
    • Per-attempt keys are the double-payout bug. Name it before they do.
    • Each call's attempt row is committed BEFORE the call goes out.
  • The ambiguous moment: we called, then crashed before recording:
    • The row stays PROCESSING.
    • On lease expiry: retry with the SAME key (the processor replays its answer), or query their status endpoint.
    • Only a definitive processor answer moves the row.
    • Retry schedule: 15m, 1h, 6h, 1d, 2d. Fits inside 5 business days with room.
    • Processor fully down? Trip a circuit breaker: leave rows PENDING, no attempt churn, and drain oldest-first on recovery.
  • Terminal states settle the escrow, in the same transaction as the status change:
    • COMPLETED → burn all of it.
    • FAILED → refund all of it.
    • PARTIAL (paid X of Y) → burn floor(robux × X/Y), refund the rest.
    • CANCELLED (only while PENDING) → refund all of it. The cancel is a compare-and-set on PENDING, so it cannot race a claim. Every terminal state settles its escrow; none may strand it.
    • Ambiguous X (their report says PARTIAL but X is missing or bigger than Y) → CAS to ON_HOLD, park in the dead-letter TABLE (a parking table, not queue infrastructure). No legs move; a human picks an ending above.
  • Reconciliation, daily, both directions:
    • Their report has a line, we have no row: repair from processor truth, page the on-call at SEV-2.
    • We say COMPLETED or PARTIAL, their report is silent: SEV-1. Until explained, that is missing money.
🔒 The invariant that audits the whole system
  • ESCROW balance = SUM of in-flight requests' Robux. Continuously checkable.
  • Every burn has a matching line in the processor's settlement report.
  • Money left the platform if and only if Robux left the economy.
TRIGGER: "how do earnings get tracked at scale"

4) How do we handle the scale asymmetry?

Bad Solution: every dashboard read sums ledger legs
  • The earnings API sums the developer's legs per game on the ledger, on every read.
  • Those reads land on the store taking hundreds to ~1k purchase writes/sec, and they spike when purchases spike.
Good Solution: scale everything for the earnings peak
  • Shard the ledger, serve dashboards from precomputed rollups, and put queues and shards in front of the payout path too.
  • Earnings reads are fixed.
  • Still wrong: the payout path, under 0.1/sec, now carries queues and shards it has no volume for.
Great Solution: size the two sides separately
  • Size the two sides, earnings and payouts, separately rather than reaching for queues everywhere.
  • The earnings ledger is the high-scale piece. Hundreds to ~1k writes/sec, sharded. It exists regardless of DevEx.
  • Earnings displays read derived rollups built off the ledger change stream (CDC: change data capture), seconds stale and labeled asOf.
  • The withdrawable balance is exact, and it is read strongly only where it gates money, at submission. It is never read from rollups, so a display gap never gates money.
  • The payout side is ONE Postgres and a worker. Keep it that way at 1000x volume.
  • Component count is earned by the split, not by the payout path.
FLASHCARDS

The five hardest probes

show all (interview mode)
Two concurrent submissions, one balance. Where exactly does the double-spend die?
Inside the escrow debit: the condition "balance minus robux stays at or above zero" is part of the UPDATE itself, under the row lock. The second transaction waits, re-evaluates against the already-debited balance, matches zero rows, and is rejected. There is no separate check step to race.
The processor paid, then you crashed before recording it.
The row is still PROCESSING with an expiring lease. The next attempt presents the SAME idempotency key (the request id), so the processor replays its recorded success instead of paying again, and we record COMPLETED late. If even that replay is lost, the daily settlement reconciliation catches the report line with no matching row and repairs from processor truth.
Why can the rate never drift between submission and processing?
Because "which rate" is not a policy, it is data: rates are immutable versioned rows, and the submission transaction writes the rate id AND the computed USD onto the request. Processing reads the request row. There is no code path that could consult a different rate.
Walk the request state machine and what each transition writes.
PENDING to PROCESSING when a worker claims it. PROCESSING to COMPLETED, PARTIAL, or FAILED on a definitive processor answer. PENDING to CANCELLED on user cancel, guarded so it cannot race a claim. PROCESSING to ON_HOLD on an ambiguous answer (DLQ, human decides). Lease expiry loops PROCESSING back to PENDING. REJECTED is born terminal at submit. Every transition is an atomic compare-and-set on the status column, committed in the same transaction as its escrow settlement legs (burn, refund, or split), so no ending can skip the money.
Would you event-source this?
It already is, where it matters: the double-entry legs and the immutable attempts table are an append-only event log, and every balance derives from them. Rebuilding ALL state only by replay would add operational cost without adding auditability. Compliance gets every change streamed to write-once archive storage with retention, satisfying the audit ask without event-sourcing the write path.
STEP 5 OF 5

Final design + levels

wrap

Final Design

  • Robux is a closed double-entry economy, and DevEx is the one audited exit.
  • Submission is one transaction: a conditional debit into escrow, with the rate version and eligibility snapshot frozen onto the request.
  • Payout runs on a leased worker that uses the requestId as the processor idempotency key.
  • Terminal states settle escrow as ledger transfers, atomic with the status change.
  • Daily reconciliation and a continuous escrow invariant keep it honest. CDC to WORM for compliance.
DevEx payouts: final architecture A purchase writes legs to RobuxLedger, whose change stream feeds EarningsRollups and WormArchive. A developer submits through DevExAPI, which checks eligibility once and commits the request, rate lock and conditional escrow debit in one PayoutStore transaction. PayoutWorker sweeps the PENDING rows, disburses through PaymentProcessor and settles escrow in the same transaction as the terminal state; Reaper returns expired leases. ReconJob diffs our rows against the processor's daily SettlementReport. Player RobuxLedger (truth) legs: -400 player, +280 dev,+120 platform - sum zerohundreds to ~1k writes/sec,shardedexists regardless of DevEx ChangeStream(CDC) EarningsRollups seconds stale, labeled asOf WormArchive write-once, for compliance Developer DevExAPI POST /v1/devex-requestsreplay + static checks EligibilityService external; snapshot stored PayoutStore (one Postgres) DevExRequests + PayoutAttempts RateVersions immutable rows Accounts+Escrow conditional debit DeadLetterTable ON_HOLD parking submission = ONE COMMIT: idempotency record +rate lock + conditional debit -> ESCROW PayoutWorker ONE logical consumercrash-safety, not throughput r14 r15 r16 Reaper expired leases PaymentProcessor external rails NotificationService email / webhook SettlementReport their truth, daily ReconJob daily, both ways 1. buys item 400 Robux 2. CDC commit-log stream 3a. folds into rollups 3b. archives 4. developer reads earnings eventual, carries asOf 5. submit request Idempotency-Key 8. 201, inputs locked rateId + usdMinor frozen 6. eligibility ONCE answer kept on the request 7. ONE COMMIT request row+rate+conditional debit 12. terminal CAS +settle escrow burn/refund/split,SAME txn 9. sweep PENDING SKIP LOCKED +lease + attempt# 13. reap expired leases PROCESSING -> back to PENDING front: oldest back the PENDING rows ARE the work queue - a table, not queue infra 11. definitive answer paid / failed / partial(X) 10. disburse USD Idempotency-Key = requestIdnever per-attempt 14. notify after COMMIT outside the money path 15. writes the daily report every real disbursement, listed 16. daily diff both directions 17. against ourrows + burns any mismatchpages: SEV-1/2 the invariant: ESCROW balance = SUM(in-flight requests' Robux); every burn matches a settlement line.money left the platform if and only if Robux left the economy. the not-so-happy paths: processor down -> circuit breaker, rows stay PENDING, drain oldest-first on recovery* crashed after calling -> lease expiry retries the SAME key* ambiguous partial -> ON_HOLD + DeadLetterTable, a human decides* report line without our row -> repair from processor truth, SEV-2* we say COMPLETED or PARTIAL, their report silent -> SEV-1: missing money

What is Expected at Each Level?

  • Mid-level candidates are expected to build a working end-to-end design (ledger, request table with statuses, processor call with retries) and know the balance check must be transactional. Hand-waving on idempotency and reconciliation is forgivable.
  • Senior candidates are expected to produce the conditional debit (the balance check inside the debit UPDATE) unprompted, use a request-scoped processor key with the crash-before-recording walkthrough, tie escrow settlement atomically to state transitions, lock the rate by foreign key, and drive the double-spend race probe (the over-withdraw dive) instead of being led.
    • Auditability: makes every payout decision replayable, by event-sourcing everything (rebuilding all state from an event log), and names what that costs (snapshots, upcasting, replay tooling).
    • Scale asymmetry: shards the earnings ledger and serves dashboards from rollups labeled asOf instead of summing legs on every read.
  • Staff candidates are expected to frame the scale split up front, freeze inputs by construction and sell that as an auditability strategy, answer event-sourcing with judgment rather than adoption, surface partials and rounding as finance-team decisions with the mechanism ready, and keep the payout path small on purpose, saying so.