← all topics Delayed payments · say it in this order 1 Problem2 Entities + API 3 High-level design4 Deep dives 5 Final + levelsreset
SAY THIS SENTENCE FIRST

"The database is the schedule. The payments table, indexed by due time, is what the schedulers read; pollers claim due rows with SKIP LOCKED and leases, and the gateway idempotency key, the paymentId, collapses every retry into at most one charge."

STEP 1 OF 5

Understanding the problem

5 min
💸 What are we building?
  • A user schedules a payment today that should execute at a future due time: "pay P $25.00 on Sept 1".
  • Ask what is being paid: an external card/bank via a gateway, or in-game currency? Assume an external gateway (the harder case); the in-game variant swaps the gateway for the wallet service and gets simpler.
  • Ask about due-time precision: is "within a minute of due" acceptable? Assume yes; sub-second scheduling is a different problem.
  • Ask whether a payment can be edited before due, or only cancelled. Assume cancel plus re-create.
  • Ask what happens on terminal failure (card declined after retries): notify and drop, or park for manual review? Assume notify + park.

Functional Requirements

  1. Users should be able to create, query, and cancel scheduled payments.
  2. The system should execute each payment at its due time, with retries and backoff on failure.
  3. A payment must be charged at most once against the gateway, no matter what crashes.
  4. Every attempt is audited in full; nothing terminal disappears silently.

Below the line (out of scope):

  • The gateway's internals, and FX.
  • Subscription billing logic. This system is the engine such logic would schedule onto: a subscription service creates rows here, keeping this engine single-shot.

Non-Functional Requirements

Do the scale math first, because it decides the architecture
  • 100M scheduled payments/day is about 1.2K executions per second on average. That is small for a partitioned Postgres.
  • But due times are human-chosen, so they spike hard: 00:00 on the 1st of the month can hold 50 to 100x an average second.
  • So at the average rate this is a correctness problem, and the only throughput story is the due-time spike.
  1. A payment executes at most once against the gateway no matter what crashes (the money-side requirement) and at least once gets attempted (the product-side requirement).
  2. Amounts are integers in minor units (cents), never floats and never DECIMAL.
  3. Trigger lateness p99 under 60 seconds, measured due_time to first attempt.
  4. API availability 99.95%.
DONE WHEN: interviewer nods at the split: average load is a correctness problem, and the only throughput story is the due-time spike.
STEP 2 OF 5

Entities + API

4 min

Defining the core entities

  • Scheduled payment: who pays, how much, when, with a lifecycle (PENDING → PROCESSING → COMPLETED / FAILED / CANCELLED).
  • Execution attempt: one try against the gateway, with its outcome; a payment has many, and the audit trail is their history.
  • Due time: when a payment becomes due, the column the schedule is read by.
  • Gateway: the external processor, reachable only through idempotency keys.
  • The database is the schedule. There is no separate queue holding truth; the payments table, indexed by due time, is what the schedulers read, and everything else is an optimization over that read.

API or system interface

POST   /v1/scheduled-payments               // payer = authenticated actor
  Header: Idempotency-Key: key              // keys scoped (actor, key)
  { "payee": P, "amount": 2500, "dueAt": "2026-09-01T00:00:00Z" }
  → 201 { paymentId, status: PENDING }
  → 409 same key, different body
  // validated: amount > 0 and ≤ cap; dueAt in the future and ≤ horizon (1y)

GET    /v1/scheduled-payments/{id}          // { status, dueAt, attempts: [...] }
GET    /v1/scheduled-payments?status=&cursor=
DELETE /v1/scheduled-payments/{id}
  → 200 CANCELLED           // won the race
  → 409 ALREADY_PROCESSING  // lost it: too late
  • Cancel succeeds only while the payment is PENDING. Once execution has claimed it, cancel returns 409 ALREADY_PROCESSING rather than pretending.
  • All times are accepted and stored in UTC; presentation converts.
DONE WHEN: you have planted the sentence: the database IS the schedule; there is no separate queue holding truth, and everything else is an optimization over reading the table.
STEP 3 OF 5

High-level design

10 min, end to end, no dives yet

1) Users should be able to create, query, and cancel scheduled payments

Create and cancel a scheduled payment: one insert, one conditional update The Client posts a scheduled payment with a scoped Idempotency-Key to PaymentAPI, which inserts one PENDING row into PaymentsDB; a unique index on (payer, key) dedupes retries. PaymentAPI answers 201. A cancel sets CANCELLED only where the row is still PENDING; if execution already claimed it, the update matches zero rows and the Client gets 409. Client PaymentAPI validates amount + horizonkey scoped to (payer, key) PaymentsDB (Postgres) index (status, due_time)the table IS the schedule 1. POST /v1/scheduled-paymentsIdempotency-Key, scoped 2. INSERT PENDING unique (payer, key) dedupes 3. 201; cancel too late? 409 ALREADY_PROCESSING 4. DELETE: CANCELLED if still PENDING too late? update matches zero rows the not-so-happy paths: same key + different body -> 409; retry with same key is safe: 201 again
  • Creating a payment is one insert with a scoped idempotency key: a unique index on (payer, key) dedupes retries, a retry with the same key is safe and returns 201 again, and the same key with a different body gets a 409.
  • The Postgres payments table, indexed by (status, due_time), is the source of truth AND the schedule itself. Everything downstream reads it.
  • Query is a read on the row plus its attempts; the user's list view rides an index on (payer, created_at).
  • Cancel is one conditional update: set CANCELLED where the row is still PENDING. If execution has already claimed the row, the update matches zero rows and the API returns 409 ALREADY_PROCESSING.

2) The system should execute each payment at its due time

Execute at due time: a scheduler marks due rows, executors charge and record The PaymentsDB payments table holds PENDING rows indexed by (status, due_time); rows whose due_time is at or before now are due. Every few seconds the Scheduler marks due rows PROCESSING and hands them to ExecutorWorkers, which charge PaymentGateway and write the outcome back. PaymentsDB - payments table PENDING rows, index (status, due_time)         23:50 23:55 00:00 00:05 00:10 now <- due_time <= now: due Scheduler runs every few seconds ExecutorWorkers call gateway, record outcome PaymentGateway external processor 1. mark due rows PENDING -> PROCESSINGin one update 4. write outcome attempt row + status:COMPLETED, FAILED or retry 2. hand off the marked rows 3. charge where this strains: two Schedulers running the same update wait on each other's row locks, so a second scheduler adds no throughput;a worker that dies mid-call leaves its row PROCESSING forever; 00:00 on the 1st arrives as one pile
  • A scheduler job runs every few seconds: one update marks the PENDING rows whose due_time has passed as PROCESSING and returns them, and the scheduler hands them to executor workers.
  • Executor workers call the gateway, then record the outcome as an attempt row and set the payment's status to COMPLETED or FAILED, or schedule a retry.
  • The PROCESSING status keeps the next sweep from picking the row up again, so on the happy path each payment is charged once.
  • Where this strains: two Schedulers running the same update wait on each other's row locks, so a second scheduler adds no throughput; a worker that dies mid-call leaves its row PROCESSING forever. The dive on triggering exactly once handles both.
  • People pick round due times, so 00:00 on the 1st arrives as one pile; the dive on midnight of the 1st handles it.

3) Failures retry with backoff, and exhausted payments park for a human

Retries: retryable failures back off, terminal ones fail, exhausted ones park ExecutorWorkers charge PaymentGateway, which answers success, a retryable failure or a terminal failure. ExecutorWorkers write an attempt row and the payment's status to PaymentsDB: COMPLETED on success, a retry after 1m, 5m, 30m, 2h or 12h with jitter on a retryable failure, FAILED on a terminal one. After about five retryable failures ExecutorWorkers park the payment in the DLQ with its attempt history for a human. PaymentsDB payment row: statusattempt rows: one per try ExecutorWorkers call gateway, record outcome PaymentGateway external processor DLQ - parked payments payment payment payment 2. write attempt row + status success: COMPLETEDretryable (network, 5xx, rate limit): retry after1m, 5m, 30m, 2h, 12h with jitterterminal (declined, account closed): FAILED 1. charge answers: success,retryable or terminal 3. budget of ~5 spent: park full attempt history attached front a human re-drives or refunds where this strains: a retry after a worker crash or a gateway timeout could charge the card a second time
  • Retry policy: exponential backoff with jitter (1m, 5m, 30m, 2h, 12h), a budget of about 5 for retryable failures (network, 5xx, rate limit).
  • Terminal failures (card declined, account closed) skip retries: straight to FAILED plus a notification.
  • An exhausted budget parks the payment in a Dead Letter Queue (DLQ, a holding pen for work that needs a human) with its full attempt history; a human tool re-drives or refunds. Nothing silently disappears, which is the audit requirement doing its job.
  • A retry after a worker crash or a gateway timeout could charge the card a second time; the dive on never charging twice handles it.

4) Every attempt is audited, and payers hear about outcomes

Audit and notifications: every change streams out of the table Every status change and attempt in PaymentsDB flows out by CDC to the Kafka topic payment-events, which feeds the derived AuditLog and the NotificationService, which tells the payer when a payment executes, fails or is parked. PaymentsDB every status change + attempt Kafka: payment-events event event event AuditLog (derived) every attempt, queryable NotificationService executed / failed / parked 1. CDC commit-log stream 2a. append 2b. feeds NotificationServicenotifies the payer front consumers are derived; the payments table stays the only truth
  • Every status change and attempt flows out via Change Data Capture (CDC) to Kafka, feeding the audit trail and the notification service. These consumers are derived; the table stays the only truth.
  • Every attempt is also its own row (payment_id, attempt, started_at, gateway_key, outcome, gateway_ref), so the audit requirement is satisfied by construction below the stream.
  • The notification service tells the payer when a payment executes, fails or is parked.
DONE WHEN: the interviewer picks a box to open. Let them steer from here.
  • If they just nod and wait: "the riskiest part is the claim query, shall I open it?"
  • Running behind: protect the claim and never-charging-twice dives; cut operations first; the polling argument can shrink to one sentence.
STEP 4 OF 5

Potential deep dives

~20 min, interviewer steers
TRIGGER: "why not a delay queue / Redis / timers"

1) Why poll a database instead of using a delay queue?

  • There are four standard ways to trigger future work, and the interviewer will expect you to have rejected three of them for stated reasons, not by default.
Bad Solution: broker delay queues as the schedule
  • The broker holds each message until its delay expires, then delivers it to a worker.
  • Brokers cap the delay horizon, and a year-out payment needs a year-out timer.
  • Cancel means tombstone-chasing a message already inside the queue, and crash recovery means reconciling queue state with table state: the queue has become a second truth store.
Good Solution: Redis ZSET or a timing wheel as an accelerator
  • Redis sorted set with score = due_time, pop everything ≤ now: fine as an accelerator; wrong as truth, because its persistence and atomic-claim story is weaker than the database it would shadow.
  • Timing wheel (an in-memory ring of time buckets): the right tool inside a process; OS timers and Kafka brokers use one. Wrong across processes and restarts at day-plus horizons.
Great Solution: the database is the schedule; poll a partitioned index
  • Every alternative to polling ends up duplicating the payments table's state into a second system, and then cancel, edit, crash recovery, and audit all become synchronization problems between the two.
  • Polling a partitioned index at a few-second interval costs almost nothing at 100M rows/day and keeps one truth: cancel is a row update, recovery is free because the row is still there.
    • Day partitions keep the sweep index tiny: pollers only touch today's partition, so the hot index covers about 100M rows, not all history; old partitions age out to cold storage after settlement.
  • Cancel, edit and audit work on the row, but a delay queue puts the trigger in a second store that cannot compare-and-swap its status. That is the answer if pushed on "why is the queue-based design worse, really": cancelling means chasing an in-flight message. Polling keeps every race on one row, which is why every race in this design has a two-line answer.
  • The moment polling actually strains, the load is due-time spikes, which smearing and pacing to the gateway solve (see the dive on midnight of the 1st) without changing the architecture.
TRIGGER: "how do you know exactly one worker fires a payment" / "two pollers at once"

2) How is a due payment triggered exactly once, across concurrent pollers and crashing workers?

  • The claim is the heart of the design: it must hand each due row to exactly one worker, survive that worker dying, and leave an audit trail of every try.
Bad Solution: one cron job, once a minute, execute due rows serially
  • It is correct at small scale, and it is where to start; say that.
  • It breaks on three things: execution time exceeding the interval (the backlog compounds), needing more than one worker (no claim semantics), and a crash mid-batch (no leases).
Good Solution: claim batches with FOR UPDATE SKIP LOCKED
-- each poller, every ~5s, on the live partition:
UPDATE payments p SET
    status = 'PROCESSING',
    lease_until = now() + interval '2 min',
    attempt = attempt + 1
FROM (
  SELECT id FROM payments
  WHERE status = 'PENDING' AND due_time <= now()
  ORDER BY due_time
  LIMIT 100
  FOR UPDATE SKIP LOCKED
) due WHERE p.id = due.id
RETURNING p.*;
  • SKIP LOCKED is the whole trick: concurrent pollers lock disjoint sets, because rows another poller holds are skipped, not waited on. No coordination service, no leader election.
  • If pushed on "two pollers sweep at the same instant, walk the rows": both run the claim; the inner SELECTs lock disjoint sets; each poller's UPDATE stamps only its own batch. No row appears in two batches, no poller waits. The serialization point is row locks, which existed anyway.
Great Solution: the claim plus leases plus an attempt-number fencing token
  • Each poller claims due rows with FOR UPDATE SKIP LOCKED, so each due row goes to exactly one worker. The claim stamps status PROCESSING, a 2-minute lease and attempt+1.
  • When a worker dies, its lease (a 2-minute claim expiry) expires and the reaper (a job that returns rows with expired leases to PENDING) compare-and-swaps the row back to PENDING, attempt count intact. The lease bounds crash damage this way, and the next sweep retries. A reclaimed row whose attempt budget is spent is parked in the DLQ instead of returned to PENDING. At any instant exactly one holder has an unexpired lease or the row is PENDING.
  • The attempt number is the fencing token: recorded on every attempt row and passed through logging, so a zombie worker whose lease expired mid-call is distinguishable in the audit trail, and its late outcome write is a compare-and-swap (CAS) against the attempt it owns, failing harmlessly if the world moved on.
  • This machinery makes triggering effectively exactly-once, not the charge. The gateway call itself is where at-most-once lives, which is the dive on never charging twice.
TRIGGER: "could the card get charged twice" / "gateway call times out" / "their report disagrees with you"

3) How do we make sure the gateway never charges twice?

  • The external side effect is the one thing no transaction of ours can roll back, so every mechanism here exists to make retries collapse.
Bad Solution: gateway idempotency key per attempt
  • Keying by attempt id makes every retry a fresh charge, which is the classic double-billing bug. Name it before the interviewer does.
Good Solution: gateway idempotency key = paymentId
  • This is the load-bearing choice: every retry of a payment, across crashes, lease expiries, and backoff, presents the same key, so the gateway collapses them into at most one charge.
    • Every payment is attempted at least once and charged at most once, which together give exactly-once effect.
  • Retries follow the backoff schedule (1m, 5m, 30m, 2h, 12h with jitter, budget ~5); terminal failures go straight to FAILED plus notification, no retries.
Great Solution: the key, plus honest handling of ambiguity, plus reconciliation
  • The gateway idempotency key is the paymentId, so every retry of a payment collapses into at most one charge.
  • Ambiguous outcomes are the hard case: a timeout after sending means the charge may or may not have happened. Never guess. Keep the payment PROCESSING, and retry with the same key (safe by the above) or query the gateway's status endpoint for that key. Only a definitive gateway answer moves the row to COMPLETED or FAILED, or schedules a retry.
  • A worker that charges the card and then dies before writing COMPLETED causes no second charge: the row stays PROCESSING, the lease expires, the reaper flips it to PENDING, the next claim retries with the same gateway key, the gateway replays its recorded success without charging again, and the new worker writes COMPLETED. The charge happened once; our record caught up. If the gateway's answer were ever lost too, the daily reconciliation (a job that diffs the gateway's settlement report against our attempts table) repairs it from settlement.
  • Every attempt is a row: (payment_id, attempt, started_at, gateway_key, outcome, gateway_ref). The daily reconciliation diffs gateway settlement against exactly this table, in both directions.
    • A charge with no COMPLETED row is our miss: repair status from gateway truth.
    • A COMPLETED row with no charge is the gateway's miss or our bug: page.
    • This closes the loop that per-request idempotency alone cannot, catching the failures neither side saw.
TRIGGER: "what does the schema look like" / "what breaks at midnight on the 1st"

4) What breaks at midnight on the 1st of the month?

payments (
  id uuid pk, payer, payee
  amount_minor bigint        -- integer cents, never DECIMAL
  due_time timestamptz, due_day partition key
  status PENDING | PROCESSING | COMPLETED | FAILED | CANCELLED
  attempt int, lease_until
  idem_actor, idem_key unique (actor, key)
) PARTITION BY due_day
  INDEX (status, due_time)       -- the sweep index
  INDEX (payer, created_at)      -- user's list view

attempts ( payment_id, attempt pk, gateway_key, outcome, gateway_ref, at )
  • Humans schedule for round times, so 00:00 on the 1st can hold up to 100x an average second. The data model that carries the design is the schema above.
  • Why Postgres: the claim needs multi-row transactional SKIP LOCKED semantics, and the schedule benefits from one strongly consistent table with real indexes. At 1.2K/s average this is one primary with partitioning, no exotics.
Bad Solution: execute the whole 00:00 pile as fast as it is claimed
  • Pollers claim everything due at 00:00 and executors call the gateway as fast as rows arrive.
  • The gateway's rate ceiling breaks first at 100x, not the sweep (an index range read) and not the claim (batched row locks).
Good Solution: pace execution to the gateway's rate limit
  • Pace execution to the gateway's rate limit. The gateway's ceiling, not our sweep, is the bottleneck, so executors call it no faster than its limit and the rest of the pile waits in the table.
  • Still wrong: the contract says "at due time", and nothing yet says how late a payment in the pile may run.
Great Solution: smear within the lateness SLO, and pace to the gateway
  • A due-time spike converts throughput shortage into bounded, monitored lateness, never into drops or duplicates.
  • Lever 1, smear: execute due payments over a window. The contract says "at due time"; delivery is due plus jitter within the lateness SLO, stated honestly.
  • Lever 2: pace execution to the gateway's rate limit. The claim batches form the pacing queue.
  • The gateway's rate ceiling sets the pace, because it breaks first: the sweep is an index range read and the claim is batched row locks.
TRIGGER: "user cancels right at the due time" / "a stale worker wakes up"

5) The races: cancel vs execute, zombie workers, edits

Bad Solution: read the status, then write it
  • Cancel reads the row, sees PENDING, and then writes CANCELLED; the claim reads the same PENDING and writes PROCESSING.
  • Both reads happen before either write, so both win: the payment executes and the user is told it was cancelled.
Good Solution: cancel and claim are both conditional updates on the status
  • Cancel is UPDATE payments SET status='CANCELLED' WHERE id=? AND status='PENDING'; the claim only takes rows that are still PENDING. Whichever runs second matches zero rows.
  • Still wrong: a worker that outlives its lease can finish its gateway call late and write its outcome after a new attempt has taken the payment over.
Great Solution: every status change is a CAS on one row, and late writes are fenced by attempt number
  • Cancel vs execute: both mutations are CAS (compare-and-swap) on the same status column of the same row, so there is no window where both win. Cancel is UPDATE payments SET status='CANCELLED' WHERE id=? AND status='PENDING'. If the claim got there first, the row is PROCESSING, cancel matches zero rows and returns 409 ALREADY_PROCESSING. If cancel got there first, the claim's inner SELECT no longer sees the row. Both orders are correct.
  • If pushed on "user cancels at 11:59:59.9 for a midnight payment": two CAS mutations race on one status column. Cancel wins → row CANCELLED, the sweep never sees it. Claim wins → cancel returns 409, truthfully. There is no interleaving where both proceed, because neither mutation is a read-then-write.
  • Zombie worker: a worker that outlives its lease cannot double-charge or overwrite status, because the gateway key and a fenced outcome write stop it. The lease expires at T, the reaper flips the row to PENDING, a new claim executes with attempt N+1 and the same gateway key; the zombie finishes its stale call late. The gateway key collapses the charge to one regardless, and the zombie's outcome write is fenced: UPDATE attempts SET outcome=? WHERE payment_id=? AND attempt=N touches only its own attempt row, never the payment's status, which only the current attempt's owner CASes.
  • Edit before due: not supported, by design; cancel (CAS-guarded) and re-create with a new key. One less mutable path through the money state machine. Say it as a choice, not a gap.
TRIGGER: "gateway is down for 6 hours" / "how do you operate this" / "what would you simplify"

6) What happens when things die, and how do we operate it?

Bad Solution: keep claiming and retrying through a gateway outage
  • Pollers keep claiming due rows while the gateway is down, so every claim burns an attempt.
  • Six hours of outage spends retry budgets, and payments with nothing wrong with them park in the DLQ.
Good Solution: circuit-break, stop claiming, drain paced
  • Circuit-break and stop claiming (leave rows PENDING rather than churning attempts); on recovery, drain the backlog oldest-first, paced to the gateway's rate limit.
  • Still wrong: nothing yet measures whether the sweep keeps up, so a stalled sweep shows up only as late payments.
Great Solution: crashes lose nothing because the schedule is rows, and one health number watches the sweep
  • A poller or executor crash loses nothing, because the schedule is rows: leases expire, the reaper returns work, attempt counts survive.
  • Alerts: oldest-due PENDING age (the single best health number: if the sweep stalls, this grows), claim throughput vs due inflow, lease-expiry rate (worker health), DLQ arrival rate, gateway error and latency, reconciliation mismatches (any nonzero pages).
  • Gateway outage: circuit-break and stop claiming (leave rows PENDING rather than churning attempts); on recovery, drain the backlog oldest-first, paced to the gateway's rate limit. Backlog size and oldest-due age are the outage dashboards.
  • Database failover: the schedule is on the primary with synchronous replication; a failover pauses execution for seconds. Claims in flight either committed (the lease holds) or did not (rows still PENDING); the state machine has no torn middle.
  • SLOs: trigger lateness p99 < 60s (due_time to first attempt); at-most-once violation count: zero, audited by reconciliation; API availability 99.95%.
  • Deploys: pollers and executors are stateless; roll freely, leases cover the gap. Partitions pre-create a week ahead by a boring cron that is itself monitored.
  • Open questions to volunteer: the poll interval (5s) and lease length (2m) are guesses; derive them from the measured execution latency distribution and the lateness SLO. The smear window is a product conversation: how late is "at midnight" allowed to be? The lever exists; the number needs an owner.
  • What I would cut first: the Kafka audit fan-out. At this scale the attempts table plus the database's own CDC to a warehouse may serve audit and notifications without a streaming tier; keep Kafka only if notification latency or consumer count demands it.
  • Multi-region: single region of record; scheduled work tolerates a brief regional failover better than most systems, since due times are minutes-grained.

Flashcards: the five hardest probes

show all (interview mode)
Two pollers sweep at the same instant. Walk the rows.
Both run the claim; the inner SELECTs lock disjoint sets because SKIP LOCKED skips rows the other holds; each poller's UPDATE stamps only its own batch. No row appears in two batches, no poller waits. The serialization point is row locks, which existed anyway.
Worker charges the card, then dies before writing COMPLETED.
Row stays PROCESSING, lease expires, reaper → PENDING, next claim retries with the same gateway key; the gateway replays its recorded success without charging again; the new worker writes COMPLETED. The charge happened once; our record caught up. If the gateway's answer were ever lost too, the daily reconciliation repairs it from settlement.
User cancels at 11:59:59.9 for a midnight payment.
Two CAS mutations race on one status column. Cancel wins → row CANCELLED, sweep never sees it. Claim wins → cancel returns 409 ALREADY_PROCESSING, truthfully. There is no interleaving where both proceed, because neither mutation is a read-then-write.
What breaks at 100x on the 1st of the month?
Not the sweep (an index range read) and not the claim (batched row locks); the gateway's rate ceiling breaks first. The claim batches become the pacing queue, lateness grows within the smear window, and the honest statement is that a due-time spike converts throughput shortage into bounded, monitored lateness, never into drops or duplicates.
Why is the queue-based design worse, really?
Because cancel, edit, and audit are row operations, and a delay queue moves the trigger into a second store that cannot CAS against the row's status: cancelling means chasing an in-flight message, and crash recovery means reconciling queue state with table state. Polling keeps every race on one row, which is why every race on this board has a two-line answer.
DONE WHEN: you can answer all five flashcards from memory before opening them.
STEP 5 OF 5

Final design + what each level is graded on

2 min

Final design

Delayed payments: final architecture The Client creates or cancels through PaymentAPI, which writes one PENDING row to PaymentsDB, the table that is the schedule. SchedulerPollers claim due rows with SKIP LOCKED and a lease, hand the batch to ExecutorWorkers, which charge PaymentGateway keyed by paymentId and write the outcome back. The Reaper returns expired leases to PENDING and parks exhausted payments in the DLQ. CDC streams every change through Kafka to AuditLog and NotificationService, and a daily ReconciliationJob diffs the gateway's settlement report against the attempts table. Client PaymentAPI validate,scope (payer, key) PaymentsDB (Postgres) partitioned by due dayindex (status, due_time)the table IS the schedule - TRUTH event event AuditLog (derived) NotificationService Reaper expired leases SchedulerPollers sweep every ~5s ExecutorWorkers call, then record PaymentGateway external processor DLQ - parked payments payment payment payment ReconciliationJob daily: settlement vs attempts 1. create/cancel + Idempotency-Key 2. one insert PENDING row 9. CDC every status + attempt Kafka: payment-events front 10a. audit 10b. notify 3. claim due rows SKIP LOCKED, lease 2min,attempt+1 6. write outcome attempt row+ status CAS 7. PROCESSING -> PENDING attempt intact;dashed arrows = periodic jobs 4. batch claimed rows 5. charge key = paymentId 8. exhausted: park after about 5 retryable failures front human re-drives or refunds; nothing silently disappears 11b. attempts table what we think happened 11a. settlement report daily diff, both directions the not-so-happy paths: worker dies mid-charge -> lease expires, reaper returns the row, the retry presents the SAME gateway key, at most one charge;gateway outage -> circuit-break, rows stay PENDING, drain oldest-first paced to the gateway ceiling; DB failover -> claims committed or didn't, no torn middle;reconciliation: charge w/o COMPLETED row -> repair from gateway truth; COMPLETED row w/o charge -> page
  • Creating a payment is one insert with a scoped idempotency key; the table, partitioned by due day and indexed by (status, due_time), is itself the schedule.
  • Pollers sweep the current partition every few seconds and claim due rows with SKIP LOCKED, stamping a lease and an attempt number; executors call the gateway with paymentId as the idempotency key, so any number of retries collapses into at most one charge.
  • The reaper reclaims expired leases from crashed workers. Executors park payments that exhaust retries in the DLQ for a human, and the reaper parks reclaimed rows that have.
  • Every status change flows via CDC to Kafka for audit and notifications, and a daily reconciliation diffs the gateway's settlement report against the attempts table, both directions.
  • The design is the one-cron-job baseline grown exactly three mechanisms: SKIP LOCKED claims, leases, and gateway idempotency keys. Nothing else was added, which is the argument that the component count is earned.

What is expected at each level

  • Mid-level candidates are expected to produce a working schedule table, a poller that executes due rows, a retry path with backoff, and a cancel endpoint, and to know amounts are integer minor units.
  • Senior candidates are expected to hand each due payment to exactly one worker, to key the gateway by paymentId because per-attempt keys double-bill, to keep the cancel race a one-row conditional update with an honest 409, and to treat ambiguous gateway outcomes as "never guess, re-ask with the same key".
    • Why poll a database: names a Redis sorted set or a timing wheel and says it can only speed up the read, never hold the schedule's truth.
    • Triggering exactly once: claims due rows in batches with FOR UPDATE SKIP LOCKED, so concurrent pollers take disjoint rows without waiting.
    • Never charging twice: uses the paymentId as the gateway idempotency key, so every retry collapses into one charge, and retries with backoff.
    • Midnight on the 1st: says the gateway's rate limit breaks first and paces execution to it.
    • The races: makes cancel and claim conditional updates on PENDING, so whichever runs second matches zero rows.
    • Operating it: circuit-breaks during a gateway outage, stops claiming, and drains oldest-first, paced to the gateway's rate limit.
  • Staff candidates are expected to open with the scale math that makes this a correctness problem, to name delay queues, Redis ZSETs, and timing wheels and decline each with reasons, to state that the midnight spike breaks the gateway ceiling first and converts into bounded lateness via smear and pacing, to close the loop with two-directional settlement reconciliation, to volunteer what they would cut (the Kafka tier) and which numbers are guesses (poll interval, lease length, smear window), and to frame the whole design as a cron job grown exactly three mechanisms.
DONE WHEN: you can close with: this is the one-cron-job baseline grown exactly three mechanisms, and nothing else was added.