"Robux is a closed double-entry economy, and DevEx is the one door out of it. A request escrows Robux atomically at submission, and what happens to that escrow, destroyed (burned) or refunded, is decided by what real dollars actually did."
Understanding the problem
4 min- Creators earn Robux when players spend in their games.
- DevEx converts earned Robux into real US dollars via an external payment processor.
- It is the only place the closed in-game economy touches real money.
Functional Requirements
- Developers should be able to see Robux earnings per game in near real time.
- Developers should be able to submit a DevEx request (a minimum cashout amount and eligibility rules apply).
- The system should convert at the rate in effect at submission, not at processing.
- Developers should be able to track request status and full history.
Below the line (out of scope):
- The eligibility rule engine itself. We consume it as a service.
- Tax documentation and reporting.
- Multi-currency rails; the processor internals.
Non-Functional Requirements
- Earnings: tens of millions of purchases/day × 3 legs each: hundreds to ~1k ledger writes/sec. Spiky.
- Payouts: thousands/day. Under 0.1/sec. Tiny but high-stakes.
- You are designing two systems with opposite profiles. Say that.
- The system should never allow over-withdrawal. Balance at submission: strongly consistent.
- The system should never lose or mutate a financial record. Every decision explainable years later.
- The system should finish payouts within 5 business days, despite processor downtime, rejections, and partial payments (paid X of a requested Y).
- The system should serve earnings reads cheaply. Eventually consistent display is fine, labeled with a freshness timestamp (asOf).
Entities + API
4 minDefining the Core Entities
- Developer Account: holds the withdrawable Robux balance and a status (ACTIVE / FROZEN).
- Earnings Ledger Entry: one leg (one half of a transfer, + or -) of a Robux movement. Double-entry. The truth.
- DevEx Request: one cash-out of Robux to USD, carrying a locked rate, an eligibility snapshot and a status.
- Exchange Rate Version: immutable (rate, valid-from) row. Requests point at one.
- Payout Attempt: one processor call and its outcome. Immutable.
- System Accounts:
DEVEX_ESCROWholds in-flight Robux.ROBUX_BURNdestroys it when a payout completes.
"Robux is a closed double-entry economy, and DevEx is the one door out. A request escrows Robux atomically at submission. Burn or refund is decided by what real dollars actually did."
API or System Interface
POST /v1/devex-requests Idempotency-Key: <key> // unique per (developer + key) pair
{ "robux": 100000 }
→ 201 { requestId, PENDING, rateId, usdMinor } // usdMinor = cents; rate + USD locked NOW
→ 422, body: { error: INSUFFICIENT_BALANCE | BELOW_MINIMUM | NOT_ELIGIBLE, reasons }
→ 409, body: { error: KEY_REUSED } // same key, different body. same key + same body: replay the stored 201
GET /v1/devex-requests?cursor= // history
GET /v1/devex-requests/:id // status + every attempt
DELETE /v1/devex-requests/:id // cancel; PENDING only; 409 after
GET /v1/earnings?game=&granularity=day // display; eventual; carries asOf
- The 201 already carries the locked rate and computed USD.
- Both are fixed inside the submission transaction. The amount can never drift after.
High-level design
10 min, end to end, no dives yet1) Developers should be able to see earnings in near real time
- Every purchase writes double-entry legs: minus player, plus developer share, plus platform cut.
- Legs sum to zero, so the economy conserves by construction.
- The earnings read sums the developer's legs per game on the ledger and returns the totals with asOf.
- The same legs sum into the developer's balance, the exact withdrawable Robux.
- Every dashboard read now competes with hundreds to ~1k purchase writes/sec on the ledger; the scale-asymmetry dive takes those reads off it.
2) Developers should be able to submit a DevEx request
- Replay check first: (developer, key) seen with matching body hash → return the stored answer, even if eligibility has since changed. Body differs → 409.
- Then the eligibility check (external call). Full result snapshotted.
- Then ONE transaction does the money work: insert the request, store the rate and USD on it, check the balance, debit the developer into ESCROW. ESCROW is the system account holding in-flight Robux.
- Two submissions racing for one balance can both pass the check; the over-withdraw dive closes the gap.
The transaction, step by step:
- Insert the request row. It is the idempotency record, the row a retry lands on.
- Read the current rate version. Compute the USD. Store both on the request.
- Check the account is ACTIVE and its balance covers the amount, then debit developer, credit ESCROW.
If the check fails (the branch of step 3):
- Frozen account or insufficient balance: one error code each.
- Mark REJECTED, commit, 422. Nothing was debited, so there is nothing to refund.
- The request row survives as the idempotency record, so a retry finds it (unique index on developer + key).
3) The rate is the one at submission
- A rate change is a NEW immutable row. Never an update.
- The request stores which rate row it used and the computed USD.
- Processing days later reads the request. It cannot see any other rate.
- The guarantee is a foreign key, not a convention.
4) Developers should be able to track status and history
- State machine: PENDING → PROCESSING → COMPLETED | PARTIAL | FAILED | ON_HOLD. Plus PENDING → CANCELLED.
- REJECTED is born terminal at submit (the balance check said no; nothing escrowed).
- A payout worker picks up a PENDING request, sets PROCESSING, calls the processor, and sets the ending status from its answer.
- A COMPLETED request burns its escrowed Robux; FAILED and CANCELLED refund it.
- Every processor call writes one immutable attempt row.
- The history endpoint returns that table verbatim. Audit trail and UI at once.
- A crash between calling the processor and recording its answer, or a status change written without its escrow legs, can pay twice or strand Robux; the processor-failure dive handles both.
Deep dives
~20 min, interviewer steers1) How do we guarantee a developer can never over-withdraw?
- The race: two submissions, one 100K balance, 80K each. Both pass app-level checks.
- Read balance, compare in code, then debit.
- Between read and write, the other request commits.
- Both pass. You paid 160K from 100K. Interviewers plant this deliberately.
- Run everything at the strongest isolation level, retry on conflict.
- Correct, but you now handle retry storms in the app.
- And you pay the maximum price for a path needing exactly one guarantee.
UPDATE accounts SET balance = balance - :robux WHERE id = :dev AND status = 'ACTIVE' AND balance - :robux >= 0
- The balance check lives inside the debit UPDATE, so check and mutation are ONE atomic statement under the row lock.
- Of two concurrent submissions, the second waits, re-checks the already-debited balance, matches zero rows, and is rejected.
- No check-then-act window exists.
- "Strongly consistent balance" here means one line of SQL, not a distributed protocol.
- The minimum-cashout check is NOT here: it is static validation at the API edge (a constant cannot be raced). The WHERE clause holds only the racy checks.
- When the debit matches zero rows, re-read the account once to say WHY: frozen account or insufficient balance, one error code each. Then mark REJECTED, commit, 422.
2) How is every payout decision auditable years later?
- Update the request in place; log what happened.
- Logs rotate. Updates destroy history.
- "Why did this pay $3,500 in 2024?" has no answer a regulator accepts.
- Rebuild ALL state only by replaying an event log.
- Fully auditable, but you adopted a whole architecture: snapshots, upcasting (migrating old stored events to new schemas), replay tooling.
- This domain can answer the question more cheaply.
- Event-source where it matters, the money, and archive the change stream everywhere else.
- The ledger legs + attempts table already ARE an append-only event log.
- Rates are immutable versions, and eligibility is snapshotted onto the request.
- Request status is a normal column, and every change streams via CDC to a write-once, read-many (WORM) archive.
3) How do we finish in 5 business days when the processor fails, rejects, or half-pays?
- The worker calls the processor, and on a timeout retries with a fresh idempotency key per attempt.
- A timeout does not mean the payment failed. If the first call paid, the retry pays again.
- Marking the timed-out request FAILED and refunding is the same bug in reverse: the developer may be paid and refunded.
- Every retry sends the requestId as the processor's idempotency key, so retries collapse to at most one disbursement.
- Still wrong:
- A worker that crashes after the call leaves the row PROCESSING with nobody to finish it.
- A status change written apart from its escrow legs can strand the Robux or settle them twice.
- Nothing catches a payment the retries missed.
- Every retry sends the processor the same idempotency key, the requestId, so retries collapse to at most one disbursement.
- An ambiguous call (a timeout, or a crash before recording the answer) stays PROCESSING until the processor gives a definitive answer, with retries at 15m, 1h, 6h, 1d, 2d inside the 5 business days.
- Each terminal state settles the escrow (the Robux held since submission) in the same transaction as the status change: COMPLETED burns it, FAILED refunds it, PARTIAL splits it.
- Daily reconciliation against the processor's settlement report, in both directions, catches what retries missed.
- Volume is thousands/day, so ONE worker is enough: leases, one processor key and reconciliation exist for crash-safety, not throughput. Saying that is the judgment signal.
- Claiming work:
- Workers sweep PENDING with FOR UPDATE SKIP LOCKED. The claim takes a lease and an attempt number and sets PROCESSING.
- A reaper flips expired-lease rows back to PENDING, the one backward transition. A definitive answer is the only way forward.
- "ONE worker" means one logical consumer; run a spare; SKIP LOCKED makes extras safe.
- Calling the processor:
- The idempotency key IS the requestId. A SECOND key, one hop out: the client's Idempotency-Key dedupes developer→us; this one dedupes us→processor. Every retry collapses to at most one disbursement.
- Per-attempt keys are the double-payout bug. Name it before they do.
- Each call's attempt row is committed BEFORE the call goes out.
- The ambiguous moment: we called, then crashed before recording:
- The row stays PROCESSING.
- On lease expiry: retry with the SAME key (the processor replays its answer), or query their status endpoint.
- Only a definitive processor answer moves the row.
- Retry schedule: 15m, 1h, 6h, 1d, 2d. Fits inside 5 business days with room.
- Processor fully down? Trip a circuit breaker: leave rows PENDING, no attempt churn, and drain oldest-first on recovery.
- Terminal states settle the escrow, in the same transaction as the status change:
- COMPLETED → burn all of it.
- FAILED → refund all of it.
- PARTIAL (paid X of Y) → burn floor(robux × X/Y), refund the rest.
- CANCELLED (only while PENDING) → refund all of it. The cancel is a compare-and-set on PENDING, so it cannot race a claim. Every terminal state settles its escrow; none may strand it.
- Ambiguous X (their report says PARTIAL but X is missing or bigger than Y) → CAS to ON_HOLD, park in the dead-letter TABLE (a parking table, not queue infrastructure). No legs move; a human picks an ending above.
- Reconciliation, daily, both directions:
- Their report has a line, we have no row: repair from processor truth, page the on-call at SEV-2.
- We say COMPLETED or PARTIAL, their report is silent: SEV-1. Until explained, that is missing money.
- ESCROW balance = SUM of in-flight requests' Robux. Continuously checkable.
- Every burn has a matching line in the processor's settlement report.
- Money left the platform if and only if Robux left the economy.
4) How do we handle the scale asymmetry?
- The earnings API sums the developer's legs per game on the ledger, on every read.
- Those reads land on the store taking hundreds to ~1k purchase writes/sec, and they spike when purchases spike.
- Shard the ledger, serve dashboards from precomputed rollups, and put queues and shards in front of the payout path too.
- Earnings reads are fixed.
- Still wrong: the payout path, under 0.1/sec, now carries queues and shards it has no volume for.
- Size the two sides, earnings and payouts, separately rather than reaching for queues everywhere.
- The earnings ledger is the high-scale piece. Hundreds to ~1k writes/sec, sharded. It exists regardless of DevEx.
- Earnings displays read derived rollups built off the ledger change stream (CDC: change data capture), seconds stale and labeled asOf.
- The withdrawable balance is exact, and it is read strongly only where it gates money, at submission. It is never read from rollups, so a display gap never gates money.
- The payout side is ONE Postgres and a worker. Keep it that way at 1000x volume.
- Component count is earned by the split, not by the payout path.
The five hardest probes
show all (interview mode)Final design + levels
wrapFinal Design
- Robux is a closed double-entry economy, and DevEx is the one audited exit.
- Submission is one transaction: a conditional debit into escrow, with the rate version and eligibility snapshot frozen onto the request.
- Payout runs on a leased worker that uses the requestId as the processor idempotency key.
- Terminal states settle escrow as ledger transfers, atomic with the status change.
- Daily reconciliation and a continuous escrow invariant keep it honest. CDC to WORM for compliance.
What is Expected at Each Level?
- Mid-level candidates are expected to build a working end-to-end design (ledger, request table with statuses, processor call with retries) and know the balance check must be transactional. Hand-waving on idempotency and reconciliation is forgivable.
- Senior candidates are expected to produce the conditional debit (the balance check inside the debit UPDATE) unprompted, use a request-scoped processor key with the crash-before-recording walkthrough, tie escrow settlement atomically to state transitions, lock the rate by foreign key, and drive the double-spend race probe (the over-withdraw dive) instead of being led.
- Auditability: makes every payout decision replayable, by event-sourcing everything (rebuilding all state from an event log), and names what that costs (snapshots, upcasting, replay tooling).
- Scale asymmetry: shards the earnings ledger and serves dashboards from rollups labeled asOf instead of summing legs on every read.
- Staff candidates are expected to frame the scale split up front, freeze inputs by construction and sell that as an auditability strategy, answer event-sourcing with judgment rather than adoption, surface partials and rounding as finance-team decisions with the mechanism ready, and keep the payout path small on purpose, saying so.
Read the four Step 3 subsections aloud once, timing yourself. If they run past 90 seconds total, cut words until they fit. Then answer the first flashcard from memory before opening it.