← all topics Ticketmaster board · say it in this order 1 Problem2 Entities + API 3 High-level design4 Deep dives 5 Final + levelsappendixreset

Design an Event Ticketing System (Ticketmaster)

Hello Interview flow, board rules · one idea per line · click a chip to mark where you are
SAY THIS SENTENCE FIRST

"Ticketing is not a throughput problem. It is one hot event, where a million people fight over 50K seats in 60 seconds. So the seat row is the truth, a guarded CAS (compare-and-set) with hold_id as a fencing token referees every sale, the seat map is a hint with its staleness declared, and a waiting room paces admission to what the booking path can carry."

STEP 1 OF 5

Understanding the problem

4 min
🎫 What are we building?
  • Users browse and search for events: a concert, a game, a show at a specific venue on a specific date.
  • They open an event and see a seat map showing which seats are still available.
  • They pick seats, hold them for a few minutes, and then pay.
  • The interesting moment is the on-sale: a hot event goes live at 10:00:00 and the whole world arrives at once.

Functional Requirements

  1. Users should be able to find events by search and by browsing.
  2. Users should be able to see which seats are available for an event.
  3. Users should be able to hold specific seats and then complete a purchase.
  4. The system should survive an on-sale spike for a hot event.

Below the line (out of scope):

  • Dynamic pricing. It changes the number on the row, not the machinery that protects the row.
  • The resale market and refunds. Both are new flows over the same seat and order model.
  • Bot detection (fingerprinting, ML scoring, fraud review).
    • What stays in scope is the one layer the fairness argument actually rests on: a queue position bound to a verified account with payment history, one position per account per event, and a challenge at join.
      • Say that split out loud as a product decision, because "bots are the real adversary" cannot be both the reason for the queue and out of scope.

Non-Functional Requirements

Do the scale math first, out loud, because it reframes the entire question
  • In aggregate this system is tiny: tens of ticket writes per second. Roughly 1M events a year is 0.03 events per second and not a number worth saying. The rate that matters is tickets: hundreds of millions a year spread evenly is tens of writes per second.
  • The distribution is the whole problem: 100K to 1M users hit ONE event within 60 seconds for about 50K seats.
    • Reads dominate writes by about 1000 to 1, and a hot event's seat map can be requested 100K times per second while its seats are being sold.
  • The seat-map fan-out is the largest number in the design, about 33M messages per second for one show. 50K seats changing state across a sale window is roughly 330 deltas per second.
    • Multiplied by 100K subscribers watching one event, that gives the 33M, or order of 1 GB/s of egress for a single show.
    • Nothing else here is within three orders of magnitude of that, so the seat map design has to answer it rather than note it.
  • So this is a contention and fairness problem concentrated on one hot event, not an aggregate throughput problem.
    • Then be precise about where the contention lands. Writes are spread over 50K rows at a few hundred commits per second, which is easy.
    • But seat popularity inside an event is heavily skewed, so the good rows see thousands of losing attempts and near-100% 409 rates while the upper deck sees none.
  • Storage is trivial: one row per seat per event, so a 50K-seat stadium is 50K rows at a few hundred bytes each, a few tens of megabytes for the biggest event in the world.
  1. A seat is sold at most once. Double-selling is the disaster case and the one thing that must never happen, because you cannot un-sell a seat to a person standing at the gate.
  2. Holds must feel instant. The hold is the thing the user is racing for, so it is a single fast write, not a workflow.
  3. The seat map may be slightly stale, but staleness must never cause a double sell. Stale reads cost a retry, never correctness.
  4. The system must degrade under on-sale load rather than collapse. Serving coarser data or a longer wait is success; timing out and retry-storming is failure.
  5. Payment must be at-most-once. Charging a card twice for one order is the second disaster case.
DONE WHEN: the interviewer has heard "contention on one hot event, skewed onto the good seats, not throughput". That one sentence licenses the waiting room, the cached seat map, and the single-row CAS later.
STEP 2 OF 5

Entities + API

4 min

Defining the Core Entities

  • Event is one performance: venue, performer, datetime, on_sale_time, and status. The on_sale_time is not decoration, it is what the waiting room schedules itself around.
  • Venue owns the seat map: sections, rows, and seat labels. It is shared across many events and is pure catalog data, so it caches forever.
  • Seat is one row per seat per event, and it is the center of the whole design. It carries status (AVAILABLE, HELD, PENDING_PAYMENT, SOLD), hold_id, hold_expires, order_id, and a price tier.
    • Every hard question in this interview is answered by an UPDATE against that one seat row.
  • Order is user, event, seat ids, total, status, and the idempotency key from the request, under a unique index on (user_id, idempotency_key). It is created inside the same transaction that flips the seats.
  • PaymentAttempt is order, processor reference, and status. It exists so that an ambiguous processor answer has somewhere honest to live.

API or System Interface

GET    /v1/events?q=&city=&date=              // search; served from the index, cacheable
GET    /v1/events/{id}                        // the event page; behind the CDN
GET    /v1/events/{id}/seats                  // → { seats[], asOf } the map plus its own staleness
POST   /v1/events/{id}/holds  { seatIds[] }   // → 201 { holdId, expiresAt } from RETURNING
                                              // → 409 SEAT_UNAVAILABLE { lostSeatIds[] } = requested minus RETURNING
POST   /v1/orders  { holdId, paymentToken }   // header: Idempotency-Key
                                              // → 202 { orderId }, or 409 HOLD_EXPIRED
                                              // → replayed 202 if that key already wrote an order
GET    /v1/orders/{id}                        // poll for CONFIRMED / FAILED
DELETE /v1/holds/{holdId}                     // release early, before the hold expires
SSE    /v1/events/{id}/updates                // seat state deltas pushed to viewers
POST   /v1/events/{id}/waiting-room           // → { position, queueToken, eta }
  • The order returns 202, not 201, because the charge happens AFTER the commit.
    • The client polls GET /v1/orders/{id}, or gets pushed the result.
    • Explaining that 202 is explaining the whole payment design in one line.
  • The seat map carries an as-of timestamp (asOf): the wall-clock instant the snapshot was read out of the seats table, not when it was cached and not when the client got it. Staleness is declared rather than hidden.
    • A client that knows its map is 800ms old can say so; a client that believes a stale map is live will blame you for the 409.
  • The hold endpoint returns 409 SEAT_UNAVAILABLE rather than a 200 with a failure body, because losing a seat race is a normal, expected outcome at an on-sale, and the client retries against fresh state.
  • On a double-click, the Idempotency-Key makes the second click's RESPONSE correct.
    • Store the key under a unique index on (user_id, idempotency_key); on conflict, look up the order already written for that key and replay the same 202 with its orderId.
    • Return 409 HOLD_EXPIRED only when the hold is gone AND no order exists for that key.
    • Skip that and the second click of a successful purchase gets told its hold expired, which is the API lying to a buyer who just bought.
  • The hold response uses RETURNING on the acquire, so expiresAt is the database's value rather than the app server's arithmetic on the app server's clock.
    • RETURNING also lets a partial multi-seat failure name WHICH seats were lost (requested ids minus the ids RETURNING gave back) instead of making the client refetch the whole map.
DONE WHEN: you have said "the seats table is the source of truth and every race in this design is an UPDATE against one seat row", and the API is on the board with the 202, the asOf, and the two 409s.
STEP 3 OF 5

High-level design

10 min, end to end, no dives yet

1) Users should be able to find events

Find events: the read path is boring on purpose A fan browses through the CDN, which asks SearchService only on a miss. SearchService queries Elasticsearch and the results flow back to the fan. Postgres (Catalog) is the truth; its WAL streams to CdcIndexer, which upserts EventDocs into the index. None of this touches the booking primary. Find events: the read path is boring on purpose search and browse are cacheable, and they never touch the booking primary Fan browsing, not buying CDN event pages plusstatic venue assets SearchService typeahead, filters bycity and date Elasticsearch denormalized EventDocsderived index, not truthrebuildable from scratch Postgres (Catalog) events, venues, seat mapsTHE TRUTH for the catalog CdcIndexer reads the WAL, thenbuilds one EventDoc 1. browses events GET /v1/events?q= 2. asks on a miss the hit ratio is high 3. queries the index match + filters 4. returns ranked EventDocs top N, venue and date 5a. returns results 200 OK to its caller 5b. serves from the edge cached at the edge, TTL seconds 6. WAL streams out CdcIndexer consumes it 7. upserts EventDocs seconds behind, and fine inside the dashed box: the read path. cached, replicated, eventually consistent. the booking path (holds, orders, payments) sits behind the waiting room on a different store. that separation IS the design. the not-so-happy paths: index stale? the event page still renders from the CDN.Elasticsearch down? fall back to a narrow Postgres query, ranking degrades, browsing survives.
  • Search runs over Elasticsearch holding denormalized event documents: name, performer, venue, city, date, price range, all in one doc so a query is one hop.
  • Event pages sit behind a CDN. The page for a hot event is requested millions of times and changes rarely, which is the definition of cacheable.
  • Postgres holds the catalog as truth, and CDC (change data capture) streams changes into the index. The index is derived and rebuildable, so nothing catastrophic happens if it is seconds behind or has to be rebuilt from scratch.
  • The point of this section is not that search is hard. It is that browsing is cacheable and boring, so it must be cleanly separated from the booking path.
    • Every read served from the CDN or the index is a read that never touches the primary you are about to fight over.

2) Users should be able to see which seats are available

See the seats: a cached snapshot, never the booking primary Viewers ask SeatMapService for the map; it reads one cached snapshot per event from SeatMapCache and answers with the snapshot's asOf. Once per short TTL, SeatMapService reads the seats from Postgres (Seats) and writes a fresh snapshot to the cache, so the booking primary sees one read per event per TTL, not one per viewer. See the seats: a cached snapshot, never the booking primary 100K viewers on one hot event share one snapshot, and every reply says how old it is Viewers 100K at once, onONE hot event SeatMapService serves the map to viewersstamps asOf on every reply SeatMapCache one snapshot per eventkey seatmap:{eventId}short TTL, carries its asOf Postgres (Seats) one row per seat per eventstatus, hold_id, hold_expires, order_idthe truth the booking path writes 1. asks for the seat map GET /v1/events/{id}/seats 2. reads the snapshot one read serves everyone 3. returns the snapshot with its asOf timestamp 4. answers 200 {seats, asOf} the client renders asOf 5. reads the seats once per event per TTL 6. writes the fresh snapshot asOf is the instant of that read the honest contract: the map is a HINT, the write at booking time is the REFEREE. a stale map costs one retry, never correctness. where it strains: everyone sees the same snapshot until the next refresh, and pushing every seat change to 100K viewers is the largest number in this design.the seat map freshness dive handles both.
  • The seat map cannot be perfectly live for 100K concurrent viewers.
    • At 100K QPS against the booking primary, at exactly the moment it is committing sales, the read path would take down the write path.
  • So a SeatMapService serves a cached snapshot of the event's seats, read from the seats table once per short TTL: one read per event per TTL, not one per viewer.
    • Every reply carries the snapshot's asOf, so the client knows how old its picture is.
  • The map is a hint, and the write at booking time is the referee.
    • A stale map costs the user one retry and can never cost correctness, because nothing is decided by what the map said.
  • Where it strains: everyone sees the same snapshot until the next refresh, and pushing every seat change to 100K viewers is the largest number in this design; the seat map freshness dive handles both.

3) Users should be able to hold seats and then buy them

Hold then buy: two requests, minutes apart The buyer holds seats through BookingAPI, which sets the seat rows in Postgres (Seats) to HELD with a hold_id and an expiry. Minutes later the buyer buys: one transaction flips the seats to SOLD and inserts the order, and the reply is 202. BookingAPI then asks the ChargeWorker to charge the card at the payment processor, the worker marks the order CONFIRMED or FAILED, and the buyer polls the order. A strip shows the seat row going from AVAILABLE to HELD to SOLD with nothing held open between the two requests. Hold then buy: two requests, minutes apart the hold is data, a seat row with an expiry, so nothing is held open while the buyer types a card number Buyer picks seats,then pays BookingAPI owns the hold andthe purchasechecks the admissiontoken first Postgres (Seats) seats: status, hold_id,hold_expires, order_idorders: one row perpurchase, with itsIdempotency-Key ChargeWorker charges the cardafter the commit Payment processor outside our system 1. holds seats POST /v1/events/{id}/holds 2. sets the seats HELD hold_id, expiry in 10 min 3. buys, minutes later POST /v1/orders, gets 202 4. one transaction seats SOLD + order row 5. asks for the charge once the sale commits 6. charges the card one call per order 7. marks the order CONFIRMED or FAILED 8. polls the order GET /v1/orders/{id} the seat row at each moment: seat 12A AVAILABLEhold_id null seat 12A HELD hold_id = H7 expires t0 + 10 min set by the hold at t0. nothing stays open until the buy at +8 min:no lock, no transaction, no connection seat 12A SOLDorder_id = O42 where it strains: two buyers can reach for one seat in the same millisecond, and a hold can lapse while its owner is still paying: the dive on selling a seat at most once.the charge after the commit can fail or time out with no clear answer: the payment dive.
  • This is the heart of the design.
  • BookingAPI owns the hold and the purchase, and both write the seats table in Postgres, the one record of who owns each seat.
  • A hold sets each requested seat to HELD with a new hold_id and a hold_expires 10 minutes out: POST /v1/events/{id}/holds returns 201 { holdId, expiresAt }, or a 409 SEAT_UNAVAILABLE naming any seat that was not available.
  • A purchase is one transaction: POST /v1/orders, minutes later, flips the held seats to SOLD and inserts the order row with its Idempotency-Key, so a sale is never half written. It returns 202 { orderId }.
    • A ChargeWorker then charges the card, and the client polls GET /v1/orders/{id} for CONFIRMED or FAILED.
  • The hold and the purchase are separate requests minutes apart, so the hold is data, a row with an expiry, not a lock or an open transaction. A buyer who walks away simply lets it expire.
  • Cancel is DELETE /v1/holds/{holdId}, which sets the seats back to AVAILABLE.
  • Where it strains: two buyers can reach for the same seat in the same millisecond, and a hold can lapse while its owner is still paying; the dive on selling a seat at most once handles both.
  • And the charge after the commit can fail, or time out with no clear answer; the payment dive handles that.

4) The system should survive the on-sale spike

The on-sale spike: a virtual waiting room in front of the event Fans join a per-event WaitingRoom before the sale opens and get back an ordered position. The Admitter lets the front fan into the booking path, in order. Inside the protected zone the admitted fan asks BookingAPI to hold seats, and BookingAPI checks the admission token and sets the seat rows in Postgres (Seats) to HELD. The on-sale spike: a virtual waiting room in front of the event 1M fans, one event, 50K seats, 60 seconds. the room orders the crowd and lets the front in. everyone moves this way as the front is admitted back: fans join here WaitingRoom (per event) 999,996 more, in order  #4 #3 #2 #1 front: admitted next Fans 1M of them, at10:00:00 sharp Admitter lets fans in from thefront of the line,in order AdmittedFan holds a valid token BookingAPI holds, then orderstoken checked first Postgres (Seats) one row per seat per eventthe truth for who ownseach seat 1. fans join the room before the sale opens 2. returns a position and a queue token 3. the front fan exits in queue order 4. lets one fan in with an admission token 5. asks to hold seats POST /v1/events/{id}/holds 6. sets the seats HELD with a hold_id and expiry inside the dashed box: the protected zone, everything behind the queue where it strains: how fast the Admitter lets fans in, and how many seats one admitted fan may hold: the on-sale dive.
  • Put a virtual waiting room in front of the event.
    • Users get a queue position and a queue token before the sale opens. The admission token comes later, when the line reaches them (A4).
  • An Admitter lets users in from the front of the line, and BookingAPI checks the admission token before it takes a hold.
  • Where it strains: how fast the Admitter lets people in, and how much one admitted user may take, decide whether the booking path survives; the on-sale dive handles both.
DONE WHEN: the interviewer picks a box to open. Let them steer from here.
  • If they just nod and wait: "the riskiest part is the hold-then-confirm boundary, shall I open how a seat is never sold twice?"
  • Running behind: protect the dives on selling a seat at most once and on the on-sale moment. The scale dive is the goalpost-move answer, do not open it unprompted.
STEP 4 OF 5

Potential deep dives

~20 min, interviewer steers
TRIGGER: "two users click the same seat" / "how do you know it is not sold twice"

1) How is a seat never sold twice?

ONE REFEREE · a guarded CAS on the seat row, with hold_id as the fencing token
  • The race is two buyers reaching for the same AVAILABLE seat in the same millisecond.
  • Plus a second, sneakier race: your hold expiring while somebody else takes the seat, and you confirming anyway.
  • There are three transitions, acquire, confirm and release, and all three are guarded UPDATEs. Compensation after a failed payment adds a fourth, guarded the same way.
  • Run these statements at READ COMMITTED, because only there is "rows affected" the verdict on who won the seat.
Bad Solution: read the seat map, check availability in the application, then write
  • Read the map, see AVAILABLE, decide in application code that the seat is free, then UPDATE.
    • The gap between the read and the write is the bug. Two users both read AVAILABLE, both decide yes, and both write. Nothing in the database ever objected.
      • Adding a transaction around it does not help unless the write itself is conditional: the read told you a fact that was already stale by the time you acted on it.
Good Solution: a distributed lock per seat in Redis
  • SETNX on seat:{event}:{seat} with a TTL, do the work, release. It works, and at small scale it works well.
    • It breaks on failover: a replica promoted mid-lease can hand the same lock to a second holder, and now two clients both believe they own the seat.
  • The deeper point to make: even a correct lock still needs the resource to fence, because the lock holder can pause (GC, network) past its TTL and write late.
    • If the seat row must check a token anyway, the lock is doing less than it appears to be doing.
  • It also adds a second store that can disagree with the first, which is exactly the coupling you spend the rest of the design avoiding.
Great Solution: the guarded CAS on the seat row, with hold_id fencing the confirm
  • The write itself decides who gets the seat: taking the hold (the acquire) is one conditional UPDATE. At READ COMMITTED, rows affected is the verdict: 1 means you won, 0 means somebody else did and you return 409.
    • There is no window between deciding and writing because the decision IS the write.
    • Name the isolation level, because at REPEATABLE READ or SERIALIZABLE the loser gets SQLSTATE 40001 instead of a zero-row result, which turns a clean 409 contract into 500s and a retry storm at exactly the wrong minute.
    • Get the mechanism right, since interviewers push here. The loser did not "find" the new state: it blocked on the row lock held by the winner.
      • When the winner committed, Postgres re-evaluated the WHERE against the newly committed version (EvalPlanQual) and skipped the row.
      • That is why the count is trustworthy without you reading anything first.
  • Expired holds are reclaimed inside that same acquire statement (OR (status='HELD' AND hold_expires < clock_timestamp())), so no reaper sits on the WRITE path.
  • The confirm carries AND hold_id=:my_hold, which answers "is this still MY hold" rather than "is somebody holding it". That one clause is what makes the deliberate transaction gap safe.
    • The SET list clears hold_id, so a SOLD row cannot be un-sold later by a release guarded on that hold. Every guard pins the state as well as the identity.
  • The confirm commits the seat flip, the order insert and an outbox row (a note that the card still needs charging) together. Rows affected must equal the seat count or the transaction rolls back, so a partial multi-seat sale cannot exist, and a worker picks up the outbox row after the commit.
  • The single-row compare-and-set (CAS) is the correctness mechanism, and any lock on top would only ever be an optimization.
    • Locks reduce wasted work; they do not decide who owns the seat.

The statements, written out.

Acquire the hold.

UPDATE seats SET status='HELD', hold_id=:new_hold, order_id=NULL,
       hold_expires = now() + interval '10 minutes'
 WHERE event_id=:e AND seat_id=:s
   AND (status='AVAILABLE'
        OR (status='HELD' AND hold_expires < clock_timestamp()))
RETURNING seat_id, hold_expires;
-- rows affected = 1 means the seat is yours; 0 means 409 SEAT_UNAVAILABLE

Confirm the sale.

UPDATE seats SET status='SOLD', order_id=:o,
       hold_id=NULL, hold_expires=NULL
 WHERE event_id=:e AND seat_id=:s
   AND status='HELD'
   AND hold_id=:my_hold          -- THE FENCE
   AND hold_expires > clock_timestamp() - interval '5 seconds';
  • Sketch this as SOLD.
    • Once payments are on the table it writes PENDING_PAYMENT instead, and the payment worker flips it to SOLD on a definitive success.
      • Why: SOLD otherwise means three different things at once.

Release on cancel. Write it out; this is where the bug lives if you only gesture at "the same shape":

UPDATE seats SET status='AVAILABLE', hold_id=NULL,
       hold_expires=NULL, order_id=NULL
 WHERE event_id=:e AND seat_id=:s
   AND status='HELD'              -- pins the STATE
   AND hold_id=:h;                -- pins the IDENTITY
now() is frozen inside a transaction
  • now() is transaction_timestamp(), frozen at transaction start.
    • In a transaction that has been open a second or three, hold_expires > now() silently grants a grace equal to however long you have been open.
    • And a multi-seat acquire under-grants the TTL by its own elapsed time.
  • So: clock_timestamp() for the two comparisons, now() for SETTING hold_expires, which keeps every seat in one hold on one expiry.
    • The confirm gets a small explicit grace (> clock_timestamp() - interval '5 seconds') while the acquire reclaims at < clock_timestamp().
      • That overlap is deliberate, and the row lock referees it: whoever gets the row first wins, and the other statement sees the state the winner left.
Name the bug before the interviewer does
  • Confirming on status='HELD' alone lets you sell a seat that a DIFFERENT customer now holds: your hold lapsed, theirs replaced it, the status is still HELD, and your UPDATE happily sells it out from under them.
    • Say the distinction in one line: status only answers "is somebody holding it". hold_id answers "is this still MY hold".
  • The mirror-image bug is the expensive one. If the confirm does not clear hold_id, a SOLD row still carries a live hold_id.
    • A release guarded only on hold_id then matches it and un-sells a paid seat: one row affected, one furious customer at the gate.
  • A guard must pin the state as well as the identity.
    • Same reason the acquire and the release both clear order_id: otherwise a FAILED order stays stamped on a seat the next customer is about to buy.
The transaction boundary, stated explicitly
  • The acquire and the confirm are NOT one transaction, and must not be.
    • They are separated by minutes of user think-time while somebody types a card number.
  • The strongest reason first: with pooled connections and stateless app servers, it is not expressible.
    • The acquire and the confirm are separate HTTP requests, minutes apart, landing on different app servers holding different connections. There is no "the" transaction to keep open.
  • And it would be an outage if you could: holding a row lock and a pooled connection across human time exhausts the pool, blocks every other buyer on that seat, and holds back vacuum.
    • Vacuum specifically, because a transaction that has written keeps its xid live and the dead tuples behind it stay non-removable.
      • Do not say it "pins a snapshot": at READ COMMITTED the snapshot is taken per statement and released, so the xid is the thing doing the damage.
  • So the hold is a lock converted into data: a row you can walk away from, visible to every server, expiring on its own without anyone holding anything.
    • That deliberate gap is exactly why the fencing check is mandatory. You gave up the lock, so you must re-prove ownership at confirm time.
  • Inside the confirm transaction the guarded seat flip goes FIRST, before the order insert and the outbox row. Putting the contended seat rows first is the right inter-table lock order: you take the hot lock, decide, and either bail immediately or hold it for the short remainder.
  • Multi-seat, and the fix that actually works.
    • Sorting your seat id array does nothing: lock order inside a multi-row UPDATE is the plan's row order, not your array order.
      • So two customers can both send ascending lists and still deadlock when one gets an Index Scan and the other a Bitmap Heap Scan visiting the same rows in the opposite direction.
  • Take the locks explicitly first, then run the guarded UPDATE:
SELECT seat_id FROM seats
 WHERE event_id=:e AND seat_id = ANY(:ids)
 ORDER BY seat_id FOR UPDATE;   -- LockRows above the scan, order guaranteed
-- then the guarded UPDATE over the same ids
  • The alternative is one UPDATE per seat issued in sorted order, which gives the same guarantee at the cost of a round trip per seat.
  • Either way, the confirm is also a multi-row UPDATE over the same rows and needs the same discipline, so it takes the same ordered lock step.
  • Have a deadlock policy, since you raised deadlocks. On SQLSTATE 40P01 the whole transaction is aborted by Postgres.
    • So retry it from the top, at most twice, with a few milliseconds of jitter; past that, return 409 and let the client pick again.
    • A deadlock is never a partial write, so retry is always safe here.
  • Dedupe seatIds server-side before the statement, or a duplicate id makes the rows-affected check fail spuriously.
  • On a double-click, the seat guard makes the second click harmless: the seat is no longer HELD under that hold, so the second confirm matches zero rows. The Idempotency-Key then makes its RESPONSE correct.
The hole in lazy reclaim, and the honest fix
  • A hold expiring is the passage of time, not a write. No row changes, so there is no WAL record, no CDC delta, and no SSE push.
    • The map shows HELD forever, and the lazy reclaim never fires because nobody reaches for a seat the map says is taken.
    • Most on-sale holds are abandoned, so this is the common path, not an edge case.
  • So keep the lazy reclaim for the write path, which is correct and keeps a reaper off the hot path.
  • And add a sweeper for the READ path: it expires abandoned holds so a real UPDATE happens and a delta is emitted.
    • Say the asymmetry plainly: the write path tolerates a dead sweeper, the read path does not. If the sweeper dies, sales stay correct and the map goes stale, which is the right way round.
  • Anything reading availability has to apply the same predicate, including the degraded view: a section count is count(*) WHERE status='AVAILABLE' OR (status='HELD' AND hold_expires < now()), not status='AVAILABLE'.
    • Otherwise the page under-reports free seats by however far behind the sweeper is.
TRIGGER: "a million people at 10:00:00" / "how do you handle the on-sale"

2) What happens at the on-sale moment?

A QUEUE, NOT RANDOM SHEDDING · ordered admission paced by in-flight concurrency
  • The load is not just large, it is concentrated: one event, one shard, one set of 50K rows, one minute.
Bad Solution: let everyone through and scale the database
  • Read replicas do not help, because this is a write contention problem, and the writes are all on one event.
  • Sharding does not help either, and say why rather than sloganeering: seats within an event are trivially shardable, so the hot event is one partition only because event_id is the shard key.
    • And event_id is the shard key because a multi-seat purchase has to be one transaction and therefore one shard.
    • Sharding contains the blast radius; it does not divide the contention.
  • What actually happens: the connection pool exhausts, p99 goes vertical, most requests time out, and every timeout becomes a retry, so offered load rises exactly when capacity falls.
Good Solution: rate limit and shed
  • Cap requests per second at the edge and 429 the rest. It does protect the database, which is why it is better than nothing.
    • But it sheds at random, so a user who arrived first has no better chance than one who arrived last.
      • That is not a fair sale, and fans experience it as a lottery run by your infrastructure.
    • It also invites a retry storm, because a 429 tells the client to try again immediately, and the client that retries hardest wins. That client is usually a bot.
Great Solution: a virtual waiting room with pre-assigned ordered positions
  • Users enter the room before the sale opens and receive a position and a queue token, so the crowd is measured and ordered before it is served.
    • The wait becomes explicit, ordered, and honest: you know you are #412,908 and roughly what that means.
  • Admission is a concurrency limiter: it keeps at most N sessions in flight in the protected zone (everything behind the queue: the booking path, its connection pool, and the primary it writes to). It is a bulkhead.
    • N comes from pool size and measured p99 by Little's law, seeded by a pre-sale capacity test, with commit rate and 409 rate as feedback that adjusts N.
    • A token bucket refilled by commit rate cannot bootstrap, because at t=0 the commit rate is zero: nobody has been admitted yet.
    • When Postgres slows, p99 rises, N falls, and the queue moves slower; nothing fails.
  • The room is the backpressure: it converts a 1M-request thundering herd into a steady stream the protected zone can serve, which is why the connection pool and the seat map cache survive.
    • The connection pool, the seat map cache, and the booking primary all see a load you chose rather than a load that arrived.
  • Even a perfect design loses the sale to denial of inventory: admitted bots hold every seat and never pay.
    • Nothing above caps how many seats one admitted session can hold, so a bot farm admitted in perfect order holds the whole stadium in seconds while every CAS legitimately returns one row and the map correctly reports sold out.
    • The fix is three caps: hold cap per identity at about 8, TTL cut to about 2 minutes during the on-sale (back to 10 afterwards), and best-available assignment instead of letting people pick freely on hot sales.
  • The position counter is itself a hot key, and a worse one than booking.
    • A million ordered positions in sixty seconds is about 17K writes/sec against one per-event structure, and the ordering is exactly what you cannot split across shards.
    • So do not promise exact ordering: either positions are approximate, or each app server draws blocks of positions and hands them out locally, which keeps ordering coarse but the writes rare.
    • And say what happens if that store dies mid-sale: admission falls back to a fixed conservative N with no position display, because a missing wait estimate is survivable and an unpaced crowd is not.
  • Be honest that fairness is a product decision: first-come order is one policy, a lottery among everyone who joined in a window is another, and the business picks.
    • If bots are in scope, the lottery is the strong answer precisely because it makes speed stop mattering: bind one position per verified account per event, require payment history, challenge at join.
    • If bots are out of scope, cut them cleanly and stop using them as the argument for the queue.
TRIGGER: "how fresh is the seat map" / "the user sees a seat that is gone"

3) How fresh is the seat map, and what does staleness cost?

STALENESS IS ADMITTED · the map is a hint, the CAS is the referee
  • 100K viewers want a picture of state that changes about 330 times a second.
    • Perfect freshness is not merely expensive here, it is the thing that would break the write path.
Bad Solution: query the database for every viewer
  • 100K QPS of seat-map reads against the booking primary, at exactly the moment it is committing sales.
    • Even if the reads themselves are cheap, they consume the connections and buffer cache the writes need, so the correctness-critical path starves to serve a picture.
Good Solution: a short-TTL cache in front of the query
  • One read per event per TTL instead of one per viewer. This is most of the win and should be said first.
    • The residual problem is uniformity: everyone sees the SAME stale snapshot, so everyone reaches for the same seats that looked free in it, and the 409 rate spikes in bursts synchronized to the TTL.
    • It also gives the client no way to know how stale its picture is, so the UI cannot be honest about it.
Great Solution: cached snapshot plus SSE deltas from CDC, with asOf exposed
  • Serve the snapshot from cache, then stream deltas over SSE fed by CDC on the seat table, so the picture converges continuously instead of stepping once per TTL.
  • Expose asOf in the payload. The client can gray out contested seats, show "updated 0.4s ago", and set expectations before the user clicks.
  • The fan-out decides the granularity: section-level counts by default, per-seat detail only for the section in view. About 330 seat transitions per second times 100K subscribers is roughly 33M messages per second and order of 1 GB/s of egress for one show. So:
    • section-level counts ("Section 112: 43 seats left") are the default view
    • per-seat detail is streamed only for the client's current section or viewport
    • deltas are coalesced on a one-second tick
    • only AVAILABLE to SOLD transitions are published
      • Suppressing HELD churn removes most of the volume and stops the stream from being a live inventory feed for scrapers.
  • Watermark the stream: the snapshot carries the CDC LSN (log sequence number, Postgres's position in its write-ahead log) it was built at, and deltas carry monotonic sequence numbers.
    • Clients drop anything at or below their watermark and resnapshot only on a detected gap.
    • Reconnects use jittered backoff against a single-flight snapshot endpoint.
    • Without that a client silently misses a delta, and "just re-GET the snapshot on reconnect" is a 1M-client stampede at the worst moment of the day.
  • An expired hold writes nothing, so CDC emits no delta for it. The read-path sweeper exists to turn abandoned holds into real writes.
  • Every availability count also has to read AVAILABLE OR (HELD AND hold_expires < now()), or the map under-reports free seats.
  • Close with the framing: the map is a hint and the CAS is the referee, so a stale map costs one retry and never costs correctness.
    • That sentence is what makes all the caching above defensible instead of sloppy.
TRIGGER: "the payment times out" / "could the card be charged twice"

4) What if payment fails or the processor times out ambiguously?

AT-MOST-ONCE · key by orderId, never guess on a timeout, compensate on definitive failure
Bad Solution: charge after the commit with no key, and guess on a timeout
  • The sell transaction commits, then a worker calls the processor, and a timeout is read as a success or a failure.
    • A timeout after sending means the charge may or may not have happened. Retry it with no key and the card can be charged twice; call it failed and release the seat, and a paid seat goes back on sale.
    • A worker that dies between the commit and the charge leaves a sold seat that is never charged, because nothing records that the charge is owed.
Good Solution: key the charge by orderId and write an outbox row in the sell transaction
  • The orderId is the processor's idempotency key on every attempt, so a retry can never become a second charge.
  • The transaction that sold the seat also writes an outbox row, so a worker dying between commit and charge leaves work that a sweeper picks up, not work that is lost.
  • What is still wrong: SOLD means three different things, paid, charge in flight, and charge ambiguous, and a definitive failure has no clean way back.
    • Releasing the seat with the hold_id guard matches zero rows, because the seat is no longer HELD and its hold lapsed long ago.
Great Solution: never guess, park the seat in PENDING_PAYMENT, and compensate by order
  • The charge is keyed by orderId, presented as the processor's idempotency key on every attempt, so a retry across crashes, restarts, and sweeps can never become a second charge.
  • The seat is already committed before the money exists, so say that uncomfortable part explicitly.
    • That is the price of not holding a lock across the processor call, and it makes compensation a designed path rather than an assumption.
  • Never guess on a timeout. A timeout after sending means the charge may or may not have happened.
    • There are exactly two honest moves: retry with the same key, or query the processor's status endpoint for that key.
    • Only a definitive answer moves the order to CONFIRMED or FAILED.
  • On a definitive failure, compensate with a guard keyed by the order, not by hold_id. Reusing the hold_id guard matches zero rows, because the seat is no longer HELD and its hold lapsed long ago.
  • Give the seat a PENDING_PAYMENT state, because SOLD currently means three different things: paid, charge in flight, and charge ambiguous.
    • The seat flips to PENDING_PAYMENT at commit and to SOLD on a definitive success.
    • order.status is the money truth and the gate scanner trusts it, not the seat row.
    • Bound how long a seat may sit unpaid during ambiguity (minutes, not hours), then force a resolution rather than leaving it parked.
  • The outbox sweeper finishes the job after a crash. The transaction that sold the seat also wrote the outbox row, so a worker dying between commit and charge leaves work that is picked up, not work that is lost.
UPDATE seats SET status='AVAILABLE', order_id=NULL,
       hold_id=NULL, hold_expires=NULL
 WHERE event_id=:e AND order_id=:o AND status IN ('SOLD','PENDING_PAYMENT');
-- then mark the order FAILED and notify
  • The good news worth saying: that guard is safe against a concurrent re-hold and is idempotent.
    • No acquire predicate matches a SOLD or PENDING_PAYMENT row, so nobody can be mid-acquire on it, and a second run of the compensation matches zero rows and does nothing.
  • Clearing order_id matters too, or a FAILED order stays stamped on a seat the next buyer is about to take.
  • The seat map handles a sold-then-released seat the way it handles everything else, as a delta.
  • Name the genuinely nasty case rather than hiding it: the processor settles the original charge after you declared failure, released the seat, and resold it.
    • The second sale is real and cannot be un-resold, so this resolves as money, not as inventory: auto-refund the first charge, detect it from the nightly settlement reconciliation, and issue a make-good.
    • Bounding the ambiguity window is what keeps this rare.
TRIGGER: "what melts first" / "how do you scale this"

5) Scale, and what melts first

SHARD BY EVENT_ID · but the first thing to break is the read path, not the database
Bad Solution: spread one hot event's seats across many shards
  • Seats inside one event are trivially shardable, so splitting a hot event across shards looks like it divides the contention.
    • But a multi-seat purchase then spans shards, so the seat flip and the order insert stop being one transaction, and an all-or-nothing sale needs a distributed transaction.
    • And it attacks the wrong number: the database is not what melts first.
Good Solution: shard by event_id
  • All the seats for one event live together, so a multi-seat purchase stays one transaction on one shard, and a hot event's contention stays inside that shard.
  • What is still wrong: it treats the database as the bottleneck, when 50K seats over a few minutes is a few hundred commits per second that one primary handles.
    • The seat map read path and the connection pool melt first, and an async standby promoted on failover can bring back seats already sold.
Great Solution: shard by event_id, defend the read path and the pool, and monitor the invariant
  • What melts first is the seat map read path and its fan-out, not the database.
    • 100K QPS of map requests arrives before the sale even opens, while the write rate is still zero.
    • Once seats start moving, 330 deltas per second to 100K subscribers is 33M messages per second if you stream per-seat to everyone.
      • That number is why section counts are the default and per-seat detail is scoped to the section in view.
  • The second thing to melt is the connection pool, because every admitted user holds a connection for the length of their request and a spike turns queueing into timeouts, which turn into retries.
    • That is exactly what the waiting room protects, and it is worth saying in those terms: the queue exists because the two things that break first are both a function of concurrent users, not of seats sold.
  • Shard by event_id. All the seats for one event live together, so the guarded CAS and the multi-seat transaction stay single-shard, and a hot event's contention is contained inside one shard instead of smeared across all of them.
  • Synchronous commit to at least one standby on the seats shards, so a failover loses no committed sale (RPO=0, a recovery point objective of zero). The whole correctness story is one Postgres row.
    • With async replication a failover promotes a standby that is missing committed sales: seats resurrect as AVAILABLE and get sold a second time, which is the one NFR that must never break.
    • The extra commit latency is affordable precisely because this design was built not to care about write throughput.
  • Instrument the invariant, not just the boxes. NFR one is "a seat is sold at most once", so:
    • run a continuous query that alarms on any seat with more than one confirmed order
    • reconcile nightly against the processor's settlement file in BOTH directions (charges with no order, orders with no charge)
    • page on outbox depth and oldest-unclaimed-age as the payment SLO
    • track 409 rate and hold-to-purchase conversion as the fairness signals
      • A conversion rate falling toward zero while holds climb is denial of inventory in progress.
  • Browsing is served from the CDN and Elasticsearch, never from the booking primary. Catalog reads go to read replicas. The hot event's seat map is served from cache.
  • Sizing sanity check: 50K seats sold over a few minutes is only a few hundred commits per second on one shard, which a single primary handles comfortably.
    • The design is not fighting write volume, it is fighting arrival concurrency.
  • State the contention story precisely, because the averages hide it. Write contention is spread across 50K rows at a few hundred commits per second, which is nothing.
    • But popularity inside one event is heavily skewed, so front-section rows take thousands of failed CASes and near-100% 409 rates while the upper deck sees essentially none.
    • The hot key is a few hundred seats inside a hot event, not the event as a whole, and that is what the queue and best-available assignment are really smoothing.
  • General admission is the real single hot key, and it is the one case where "a single hot key" is literally true.
    • GA has no seat rows, just a quantity, so the CAS collapses to one counter row that every buyer serializes on.
    • Fix it by bucketing: split the inventory into K counter rows, hash each buyer to a bucket and fall back to scanning the others when theirs is empty.
      • Reconcile the last few units under a single-bucket fallback, where contention is trivially low because almost nothing is left.
FLASHCARDS · THE FIVE HARDEST PROBESshow all (interview mode)
Two users click the same seat in the same millisecond. Walk me through exactly what happens.
  • Both clients POST /holds with the same seatId, and both requests run the same conditional UPDATE: SET status='HELD', hold_id=:new_hold, order_id=NULL, hold_expires=now()+10min WHERE event_id=:e AND seat_id=:s AND (status='AVAILABLE' OR (status='HELD' AND hold_expires < clock_timestamp())) RETURNING hold_expires.
    • We are at READ COMMITTED, which is what makes rows-affected a verdict at all.
  • Postgres serializes them on the row: the first commits, flips the row, and gets rows affected = 1, so it returns 201 with a holdId and the database's own expiry.
  • The second did not "find" the new state, it blocked on the row lock.
    • When the winner committed, Postgres re-evaluated the WHERE against the newly committed version (EvalPlanQual) and skipped the row, so rows affected = 0 and it returns 409 SEAT_UNAVAILABLE.
    • Worth naming the isolation level explicitly: at REPEATABLE READ or SERIALIZABLE the loser would get SQLSTATE 40001 instead, turning a clean 409 into a 500 and a retry storm.
  • There is no gap between deciding and writing because the decision IS the write, and no application code ever compares a read to a state it hopes is still true.
  • The loser's client refetches the seat map (which by then has the delta over SSE) and picks another seat.
  • Note what did NOT happen: no lock service, no coordination, no distributed transaction. The row is the referee.
A user's hold expires while they are on the payment page. What do they see, and what stops you from selling that seat twice?
  • They see a 409 HOLD_EXPIRED and their card is not charged.
  • Mechanically: the confirm is an UPDATE ... SET status='SOLD', order_id=:o, hold_id=NULL, hold_expires=NULL WHERE ... AND status='HELD' AND hold_id=:my_hold AND hold_expires > clock_timestamp() - interval '5 seconds'.
    • If their hold lapsed and another buyer re-acquired the seat, the row now carries a different hold_id (and the acquire statement reclaimed it in the same breath).
    • So the guard matches zero rows, the whole transaction rolls back, and no order, no outbox row, and no charge exist.
  • Two details that are easy to drop and both bite.
    • One: the SET list must CLEAR hold_id, or a SOLD row keeps a live hold_id and a release guarded only on hold_id will un-sell a paid seat, which is why every guard pins the state as well as the identity.
    • Two: the comparison uses clock_timestamp(), because now() is transaction_timestamp() and would silently grant a grace equal to however long the transaction has been open.
  • This is also why hold_id is a fencing token rather than decoration: confirming on status='HELD' alone would have sold the seat that a DIFFERENT customer now holds.
    • Status only answers "is somebody holding it" while hold_id answers "is this still MY hold".
  • The gap is deliberate, and with pooled connections and stateless app servers it is not even optional: the acquire and the confirm are separate HTTP requests landing on different servers and different connections, so there is no single transaction to hold open.
    • And if there were, it would pin a live xid, hold back vacuum, and block every other buyer on that seat.
  • The hold is a lock converted into data, and the fence is the price of that conversion.
A million people hit the on-sale at 10:00:00. What actually falls over first?
  • Not the database. The first thing to break is the fan-out on the read path, and it is worth computing out loud.
    • 50K seats transitioning across the sale window is about 330 deltas per second, and 100K subscribers on one event makes that roughly 33M messages per second and order of 1 GB/s of egress for one show.
      • So section counts are the default, per-seat detail is scoped to the client's current section, deltas are coalesced on a one-second tick, and only AVAILABLE to SOLD transitions are published.
    • Even before the sale opens, 100K QPS of map requests arrives while the write rate is still zero, so those reads are served from a cached snapshot with an asOf, never from the primary.
  • The second thing to break is the connection pool, because every admitted user holds a connection for the length of their request.
    • Queueing turns into timeouts and every timeout becomes a retry, so offered load rises as capacity falls.
  • Both failures are functions of concurrent users, not of seats sold, which is exactly what the waiting room addresses.
    • And it does it as a CONCURRENCY limiter: hold N sessions in flight in the protected zone, N derived from pool size and measured p99 by Little's law and adjusted by commit rate and 409 rate.
      • Pacing on commit rate alone cannot bootstrap, since at t=0 nobody has been admitted and the commit rate is zero.
  • The database itself is comparatively fine: 50K seats over a few minutes is a few hundred commits per second on one shard, though popularity skew means the good rows still see near-100% 409 rates.
  • And handle the shard question precisely: seats within an event are trivially shardable, so the hot event is one partition only because event_id is the shard key, and event_id is the shard key because a multi-seat purchase must stay one transaction and therefore one shard.
The payment processor times out and you do not know whether the card was charged.
  • Never guess. The order stays in its pending state and we do exactly one of two things: retry with the same key, or query the processor's status endpoint for that key.
    • The key is orderId, so a retry can never become a second charge, and only a definitive answer moves the order to CONFIRMED or FAILED.
  • If the worker crashed instead of timing out, the outbox row written inside the sell transaction is still there, and the sweeper picks it up and finishes charge-and-deliver.
  • On a definitive failure we compensate, and the guard has to change: releasing on hold_id would match zero rows, because the seat is no longer HELD and its hold lapsed long ago.
    • So the statement is UPDATE seats SET status='AVAILABLE', order_id=NULL, hold_id=NULL, hold_expires=NULL WHERE event_id=:e AND order_id=:o AND status IN ('SOLD','PENDING_PAYMENT'), then mark the order FAILED.
    • That guard is safe against a concurrent re-hold and idempotent, since no acquire predicate matches a sold row and a second run affects nothing.
  • The uncomfortable part to say out loud is that the seat is already committed before the money exists, because we deliberately do not hold anything across the processor call.
    • Which is also why the seat sits in PENDING_PAYMENT while the charge is in flight, and order.status stays the money truth the gate scanner trusts.
    • So compensation is a designed path, not an assumption, and the seat map treats a sold-then-released seat as just another delta.
  • The genuinely nasty case is the processor settling the original charge AFTER we declared failure and resold the seat.
    • That one cannot be fixed in inventory, so it resolves as money: an auto-refund caught by the nightly two-way settlement reconciliation plus a make-good.
Why not just use Redis for the hold?
  • Because it would not be doing the job it looks like it is doing.
    • A Redis lock per seat works until failover, when a promoted replica can hand the same lock to a second holder.
    • And even a perfectly correct lock still needs the resource to fence, since a holder can pause past its TTL and write late.
      • If the seat row has to check a token anyway, the lock is an optimization, not the correctness mechanism.
  • Meanwhile the seat row already gives us a conditional UPDATE whose rows-affected count is an atomic verdict, in the same store and the same transaction as the order insert and the outbox row, so the sale is all-or-nothing without a distributed transaction.
  • Adding Redis would add a second store that can disagree with the first, and reconciling a lock state with a row state is the coupling this whole design exists to avoid.
  • Redis is a fine place for the waiting room queue and the seat map cache, which are both things that can be rebuilt.
    • It is the wrong place for the fact that decides who owns a seat.
DONE WHEN: you can write all three UPDATE statements from memory, including the OR-reclaim clause, the hold_id fence, the SET that clears it, and the release guarded on status AND hold_id, and the dives on selling a seat at most once, the on-sale moment, and seat map freshness ended at their Great card with the alternative named and declined.
STEP 5 OF 5

Final design + what is expected at each level

wrap

Final Design

Final design: three paths, one referee The fan browses through the CDN, SearchService and an Elasticsearch index that CdcIndexer builds from the catalog. The seat map comes from a cached snapshot, kept moving by deltas that DeltaPublisher pushes over SSE from the seats CDC feed. To buy, the fan joins the WaitingRoom; the Admitter lets fans into BookingAPI, which runs the guarded CAS on Postgres (Seats) and writes the outbox row in the same commit, and ChargeWorker charges the external PaymentProcessor once per orderId. Final design: three paths, one referee browse is cached and derived. the seat map is a hint with an asOf. the booking path is guarded by one CAS on one row. Fan one person, threedifferent paths CDN event pages SearchService typeahead and filters Elasticsearch EventDocs, derived Postgres (Catalog) events, venues, seat maps CdcIndexer catalog -> index SeatMapService asOf on every reply SeatMapCache one snapshot per event DeltaPublisher deltas from CDC WaitingRoom ordered positionsadmission tokens  #2 #1 Admitter concurrency limiterN in-flight sessions BookingAPI holds, orders,idempotency key,the guarded CAS Postgres (Seats) sharded by event_idone row per seatPENDING_PAYMENT while chargingorder.status is the money truth Outbox written in the SAME tx  orderId orderId orderId ChargeWorker charge keyed by orderIdsweeper finishes crashes PaymentProcessor external, at-most-once dashed = outside our system 1. browses events 2. asks on a miss 3. queries 4. upserts EventDocs 5. streams changes 6. opens the map 7. reads snapshot 8. pushes seat deltas over SSE the map is a hint, the CAS is the referee 9. joins the room before on-sale 10. front exits 11. admits, paced 12. guarded CAS 13. writes the outbox row inside the SAME commit 14. claims work from the outbox 15. charges once key = orderId 16. CDC feeds the deltas one sentence: browse from the CDN and the index, read seats from a cached snapshot with an asOf, and let ONE guarded CAS on the seat row referee every sale. the not-so-happy paths: hold expires? the next acquire reclaims it in the same statement. crash after the commit? the outbox sweeper finishes charge-and-deliver.processor ambiguous? never guess: retry the same key or query its status, and on a definitive failure release the seats and mark the order FAILED.
  • Browsing is cached and derived: CDN in front of event pages, Elasticsearch for search, catalog truth in Postgres with CDC feeding the index.
  • The seat map is a cached snapshot carrying an asOf and the CDC LSN it was built at, kept moving by sequence-numbered SSE deltas, section counts by default and per-seat detail only for the section in view.
  • The waiting room holds ordered positions and admits by concurrency, keeping at most N sessions in flight on the booking path behind it, which makes it backpressure rather than a UI feature. Per-identity hold caps and a short on-sale TTL stop an admitted bot farm from holding the stadium.
  • The booking path is two transactions on purpose. First a guarded acquire that also reclaims expired holds on the write path.
  • Then a guarded confirm fenced by hold_id that flips the seats, clears hold_id, inserts the order, and writes the outbox row in one commit, seats first.
  • A read-path sweeper expires abandoned holds so the map gets a delta, since an expiry by itself writes nothing. The write path survives a dead sweeper; the read path does not.
  • The charge happens after the commit, keyed by orderId, with the seat in PENDING_PAYMENT until the answer is definitive, a sweeper for crashes, and a compensation path guarded on order_id.
  • Seats are sharded by event_id, so a multi-seat purchase stays one transaction on one shard, and the seats shards commit synchronously to a standby so a failover cannot resurrect sold seats.
  • The invariant is monitored, not assumed: a continuous query alarms on any seat with more than one confirmed order, and settlement reconciles both ways nightly.

What is Expected at Each Level

  • Mid-level candidates are expected to produce the seat, hold and order model, and to know that the write must be guarded rather than decided in application code after a read.
  • Senior candidates are expected to:
    • nail the hold_id fencing token AND clear it on the confirm
    • guard every transition on state as well as identity
    • name the isolation level that makes rows-affected a verdict
    • state the transaction boundary explicitly (with pooled connections it is not even expressible, and it would hold back vacuum if it were)
    • use an outbox so the charge happens after the commit, at-most-once in effect (the outbox delivers at least once, the idempotency key at the processor collapses the retries)
    • on the on-sale moment, go past scaling the database to admission at the edge, and say why rate limiting that sheds at random is unfair and invites a retry storm
    • on seat map freshness, serve the map from a short-TTL cache, one read per event instead of one per viewer, and say that everyone then sees the same stale snapshot
    • on scale, shard by event_id so a multi-seat purchase stays one transaction on one shard
  • Staff candidates are expected to:
    • open with the concentration math that reframes the problem from throughput to skewed contention inside one hot event
    • compute the SSE fan-out and let that number pick the seat map design rather than noting it as an aside
    • treat the waiting room as a concurrency limiter rather than a UI feature, and see the denial of inventory it does not prevent
    • monitor the at-most-once invariant instead of asserting it
    • name honestly what the design does not solve: the processor that settles a charge after the seat was resold, and bots as the actual adversary behind every fairness mechanism
DONE WHEN: you can point at the final diagram and name, in one breath, the guarded CAS, the hold_id fence, the deliberate gap, the outbox, and the waiting room as backpressure.
APPENDIX A

How the Admitter actually processes the queue

reference, not one of the 5 steps
TRIGGER: "you said concurrency limiter. how does it actually admit people?"
ONE MOVING LINE · not a pop per user
The Admitter: it moves one line, it never walks the queue The waiting line holds fan positions; one number, admitted_through, marks how far the sale has reached. Each tick the Admitter counts the live leases in LeaseSet and advances that number with one INCRBY. Fans see the line pass, show up at BookingAPI with a signed token, and BookingAPI opens a lease inside the protected zone. The Admitter: it moves one line, it never walks the queue admitting 5,000 fans is one INCRBY. the fans compare themselves to the line: nothing is popped, nobody is pushed. #412,911 #412,910 #412,909 #412,908 #412,907 #412,906 back: 1M fans joined here front: the sale is through here the line sweeps backwards; the fans never move admitted_through = 412,907 still waiting, holding an SSE stream past the line: may enter now past the line you have about 60s to show up,or the admission is void and you rejoin at the back Fans already have a positionand an SSE stream open my position: 412,908 the line is at 412,907so: not yet. 1 to go. Admitter (per event) one tick every 500ms no leader election needed:the read and the INCRBY areone atomic Lua call claim_rate = of the positions the line passedin the last 30s, the share that actually showed up.a 40% no-show rate runs the zone at 60% of N. one tick, and it must be ONE round trip -- KEYS: inflight, admitted_through   ARGV: now, N, claim_rate local live  = redis.call('ZCOUNT', KEYS[1], ARGV[1], '+inf')local slots = tonumber(ARGV[2]) - liveif slots <= 0 then return 0 endlocal advance = math.floor(slots / tonumber(ARGV[3]))return redis.call('INCRBY', KEYS[2], advance) -- two admitters cannot both see the same free capacity the protected zone: never more than N sessions inside BookingAPI writes every lease(both ZADD calls)N came from pool size and p99 LeaseSet (one ZSET per event) score = when this session's lease expires     a live tab re-ZADDs a later score. a dead tab does not,and its slot returns with nobody telling you. expired already swept now live leases = in_flight ZCOUNT now +inf 1. counts the live leases ZCOUNT inflight, now to +inf 2. advances the line byslots / claim_rate one INCRBY, whatever the depth 3. the fan sees the line pass SSE push, or the client's next poll 4. shows up with a signed token the edge verifies the HMAC locally,no Redis lookup per request 5. opens a lease on admission ZADD inflight now+120s sid the not-so-happy paths: the admitter loop dies? nothing errors. the sale just stops and every dashboard stays green,so alarm on: positions admitted in the last 30s = 0 while queue depth > 0.Redis dies? fall back to a fixed conservative N and stop showing positions at all.claim_rate too high starves the zone, too low overfills it, so measure it, do not guess.

A1) The mechanism

  • The Admitter, the waiting room's admission loop, does not walk the queue: it advances one number per event, admitted_through.
    • admitted_through is a single integer per event, and everyone whose position is at or below it may enter.
    • Admitting 5,000 fans is therefore one INCRBY, not 5,000 messages.
    • Clients hold an SSE stream or poll, and ask one question: has the line passed my position yet.
    • The alternative is popping a million entries and notifying each one, which rebuilds the thundering herd inside the thing that exists to prevent it.
  • The fans hold still and the line sweeps past them.
    • Step 3 of the diagram draws the crowd shuffling forward, which is the right mental model to say out loud; the implementation inverts it.
  • The Admitter runs one tick every 500ms, and the tick has to be one round trip, for a reason in A5.
local in_flight = redis.call('ZCOUNT', inflight, now, '+inf')  -- the lease set, see A3
local slots = N - in_flight                               -- how much room the zone has
if slots <= 0 then return 0 end

local adv   = math.floor(slots / claim_rate)              -- see A2
return redis.call('INCRBY', admitted_through, adv)      -- the whole admission

A2) Why the division by claim_rate

  • A large share of queue positions are dead: closed tabs, people who gave up, duplicate joins.
    • Advance by exactly slots and a 40% no-show rate runs the protected zone at 60% of N, so the sale takes almost twice as long for no reason.
  • So measure the share of positions that show up, claim_rate, rather than guessing it.
    • Of the positions the line passed in the last 30 seconds, what share actually exchanged a token for a session inside the grace window (defined two bullets down).
    • That is claim_rate, and dividing by it is a feedback loop that self-corrects when the crowd's behaviour changes mid-sale.
    • Too high starves the zone, too low overfills it, which is the same failure as getting N wrong.
  • The grace window is the other half: an admitted fan has about 60 seconds to show up.
    • Once the line passes you, you have about 60 seconds to actually show up and trade the position for a session.
    • Miss it and the admission is void and you rejoin at the back, because without expiry a ghost holds capacity forever.

A3) How a slot ever frees, which is the part usually gotten wrong

  • A session ends three ways, and only one of them can tell you about itself.
    • Clean exit: checkout finishes or the user leaves, so an explicit release is possible.
    • Abandoned tab: nothing is sent, ever.
    • Your own server dies while holding the count.
  • So the in-flight count cannot be a counter you increment and decrement: each admitted session is a lease that expires.
    • One sorted set per event, scored by when that session's lease expires:
      • admit is ZADD inflight <now+120s> <sessionId>
      • heartbeat is the same ZADD with a fresh score, sent by the live tab
      • in_flight is ZCOUNT inflight <now> +inf, one cheap read
      • the sweep is ZREMRANGEBYSCORE inflight -inf <now>
    • A dead tab stops heartbeating, its lease expires, and the slot returns with nobody telling you.
    • ZADD is an upsert, which is the only reason the heartbeat can be the same command.
      • A sorted set holds unique members, so an existing member cannot be added twice.
      • Member already there: its score is overwritten in place.
      • Member not there: it is added.
      • The reply counts NEW members only, so a successful heartbeat replies 0.
  • Each session is a member of one Redis sorted set scored by its lease expiry, so the sweep frees every slot whose score is below now. The walkthrough starts with three sessions, member on the left and expiry score on the right:
before            sess_A  120
                  sess_B  125
                  sess_C  130

run               ZADD inflight 180 sess_A     sess_A is already a member
reply             0                            = zero NEW members added

after             sess_B  125
                  sess_C  130
                  sess_A  180                  score 120 -> 180, so it re-sorted to the end
                                               still 3 members, no duplicate


before            sess_A  120
                  sess_B  125
                  sess_C  130

run               ZADD inflight 200 sess_D     sess_D is not a member yet
reply             1                            = one NEW member added

after             sess_A  120
                  sess_B  125
                  sess_C  130
                  sess_D  200                  appended, now 4 members
  • Say out loud that this is the seat hold one level up.
    • Same expiry-and-reclaim shape as hold_expires on the seat row, applied to admission instead of inventory.
    • Two instances of one pattern reads far better than two unrelated tricks.

A4) Who calls what, and where the token comes from

  • Redis is never reachable from the browser. Every command in A3 is server side.
  • BookingAPI, the service behind the queue, writes the leases: it makes both ZADD calls.
    • On admission: the browser POSTs /enter with the token, BookingAPI verifies it, then writes the lease.
    • On heartbeat: the browser POSTs /heartbeat every 30s, BookingAPI writes the same lease with a later score.
    • The browser's whole job is to keep pinging. It never speaks to Redis.
  • The Admitter reads and sweeps the leases and never writes one.
    • ZCOUNT on each 500ms tick, which is where in_flight comes from.
    • ZREMRANGEBYSCORE on the same tick, housekeeping only.
    • The one thing it writes is admitted_through.
  • There are two tokens, easy to conflate: the queue token proves your position, the admission token proves you may enter now.
    • Queue token: handed out at join time, before the sale. It proves "I hold position 412,908".
    • Admission token: handed out when the line reaches that position. It proves "I may enter now".
  • WaitingRoom signs the admission token and pushes it over SSE once the line passes the fan's position:
    • joins before the sale, and WaitingRoom returns a position plus a queue token
    • the browser holds an SSE stream asking whether the line has reached that position
    • the Admitter advances admitted_through past it
    • WaitingRoom signs an admission token and sends it down that stream
    • the browser POSTs /enter with it, and BookingAPI writes the first ZADD
  • The admission token carries event id, position, session id, and an expiry of about 60s, which IS the grace window from A2.
    • It is bound to one account, so it cannot be resold or handed around a bot farm.
    • HMAC signed by WaitingRoom and verified at the edge against a shared secret, so there is no lookup per request.

A5) Two admitters must not both see the same free capacity

  • If two replicas each read in_flight = 4,200 against N = 5,000, they both admit 800 and the zone overshoots by 800 sessions.
  • Do the read and the advance in one Lua script so the count and the INCRBY cannot interleave.
    • That beats electing a leader per event: no election to run, and an admitter dying mid-tick costs one tick instead of a failover.
  • Same instinct as the signed token in A4: keep the per-event hot key from coming back, whether the pressure is one tick or one request.

A6) The failure nobody notices, and what the fan sees

  • If the tick loop stops, nothing errors.
    • Postgres is healthy, the pool is empty, every dashboard is green, and the sale is dead because nobody is being let in.
    • So alarm on the invariant: positions admitted in the last 30 seconds is zero while queue depth is above zero. Zero is the alarm, not the quiet.
  • If Redis dies, admission falls back to a fixed conservative in-flight limit N with no position display, because a missing wait estimate is survivable and an unpaced crowd is not.
  • The ETA is (my_position - admitted_through) / admission rate, where the rate is an EWMA (exponentially weighted moving average) over the last 30 seconds.
    • Label it approximate, and never let it jump upward on screen if you can avoid it, because an ETA that grows is a support ticket.
    • The position counter is itself the hot key from the on-sale dive, so blocks of positions are drawn per app server and exact ordering is not promised.
DONE WHEN: you can say "it advances admitted_through and counts live leases" and then defend both halves: why the division by claim_rate exists, and why in_flight is a lease set rather than a counter.