Design an Event Ticketing System (Ticketmaster)
"Ticketing is not a throughput problem. It is one hot event, where a million people fight over 50K seats in 60 seconds. So the seat row is the truth, a guarded CAS (compare-and-set) with hold_id as a fencing token referees every sale, the seat map is a hint with its staleness declared, and a waiting room paces admission to what the booking path can carry."
Understanding the problem
4 min- Users browse and search for events: a concert, a game, a show at a specific venue on a specific date.
- They open an event and see a seat map showing which seats are still available.
- They pick seats, hold them for a few minutes, and then pay.
- The interesting moment is the on-sale: a hot event goes live at 10:00:00 and the whole world arrives at once.
Functional Requirements
- Users should be able to find events by search and by browsing.
- Users should be able to see which seats are available for an event.
- Users should be able to hold specific seats and then complete a purchase.
- The system should survive an on-sale spike for a hot event.
Below the line (out of scope):
- Dynamic pricing. It changes the number on the row, not the machinery that protects the row.
- The resale market and refunds. Both are new flows over the same seat and order model.
- Bot detection (fingerprinting, ML scoring, fraud review).
- What stays in scope is the one layer the fairness argument actually rests on: a queue position bound to a verified account with payment history, one position per account per event, and a challenge at join.
- Say that split out loud as a product decision, because "bots are the real adversary" cannot be both the reason for the queue and out of scope.
- What stays in scope is the one layer the fairness argument actually rests on: a queue position bound to a verified account with payment history, one position per account per event, and a challenge at join.
Non-Functional Requirements
- In aggregate this system is tiny: tens of ticket writes per second. Roughly 1M events a year is 0.03 events per second and not a number worth saying. The rate that matters is tickets: hundreds of millions a year spread evenly is tens of writes per second.
- The distribution is the whole problem: 100K to 1M users hit ONE event within 60 seconds for about 50K seats.
- Reads dominate writes by about 1000 to 1, and a hot event's seat map can be requested 100K times per second while its seats are being sold.
- The seat-map fan-out is the largest number in the design, about 33M messages per second for one show. 50K seats changing state across a sale window is roughly 330 deltas per second.
- Multiplied by 100K subscribers watching one event, that gives the 33M, or order of 1 GB/s of egress for a single show.
- Nothing else here is within three orders of magnitude of that, so the seat map design has to answer it rather than note it.
- So this is a contention and fairness problem concentrated on one hot event, not an aggregate throughput problem.
- Then be precise about where the contention lands. Writes are spread over 50K rows at a few hundred commits per second, which is easy.
- But seat popularity inside an event is heavily skewed, so the good rows see thousands of losing attempts and near-100% 409 rates while the upper deck sees none.
- Storage is trivial: one row per seat per event, so a 50K-seat stadium is 50K rows at a few hundred bytes each, a few tens of megabytes for the biggest event in the world.
- A seat is sold at most once. Double-selling is the disaster case and the one thing that must never happen, because you cannot un-sell a seat to a person standing at the gate.
- Holds must feel instant. The hold is the thing the user is racing for, so it is a single fast write, not a workflow.
- The seat map may be slightly stale, but staleness must never cause a double sell. Stale reads cost a retry, never correctness.
- The system must degrade under on-sale load rather than collapse. Serving coarser data or a longer wait is success; timing out and retry-storming is failure.
- Payment must be at-most-once. Charging a card twice for one order is the second disaster case.
Entities + API
4 minDefining the Core Entities
- Event is one performance: venue, performer, datetime, on_sale_time, and status. The on_sale_time is not decoration, it is what the waiting room schedules itself around.
- Venue owns the seat map: sections, rows, and seat labels. It is shared across many events and is pure catalog data, so it caches forever.
- Seat is one row per seat per event, and it is the center of the whole design. It carries status (AVAILABLE, HELD, PENDING_PAYMENT, SOLD), hold_id, hold_expires, order_id, and a price tier.
- Every hard question in this interview is answered by an UPDATE against that one seat row.
- Order is user, event, seat ids, total, status, and the idempotency key from the request, under a unique index on (user_id, idempotency_key). It is created inside the same transaction that flips the seats.
- PaymentAttempt is order, processor reference, and status. It exists so that an ambiguous processor answer has somewhere honest to live.
API or System Interface
GET /v1/events?q=&city=&date= // search; served from the index, cacheable
GET /v1/events/{id} // the event page; behind the CDN
GET /v1/events/{id}/seats // → { seats[], asOf } the map plus its own staleness
POST /v1/events/{id}/holds { seatIds[] } // → 201 { holdId, expiresAt } from RETURNING
// → 409 SEAT_UNAVAILABLE { lostSeatIds[] } = requested minus RETURNING
POST /v1/orders { holdId, paymentToken } // header: Idempotency-Key
// → 202 { orderId }, or 409 HOLD_EXPIRED
// → replayed 202 if that key already wrote an order
GET /v1/orders/{id} // poll for CONFIRMED / FAILED
DELETE /v1/holds/{holdId} // release early, before the hold expires
SSE /v1/events/{id}/updates // seat state deltas pushed to viewers
POST /v1/events/{id}/waiting-room // → { position, queueToken, eta }
- The order returns 202, not 201, because the charge happens AFTER the commit.
- The client polls GET /v1/orders/{id}, or gets pushed the result.
- Explaining that 202 is explaining the whole payment design in one line.
- The seat map carries an as-of timestamp (
asOf): the wall-clock instant the snapshot was read out of the seats table, not when it was cached and not when the client got it. Staleness is declared rather than hidden.- A client that knows its map is 800ms old can say so; a client that believes a stale map is live will blame you for the 409.
- The hold endpoint returns 409 SEAT_UNAVAILABLE rather than a 200 with a failure body, because losing a seat race is a normal, expected outcome at an on-sale, and the client retries against fresh state.
- On a double-click, the Idempotency-Key makes the second click's RESPONSE correct.
- Store the key under a unique index on (user_id, idempotency_key); on conflict, look up the order already written for that key and replay the same 202 with its orderId.
- Return 409 HOLD_EXPIRED only when the hold is gone AND no order exists for that key.
- Skip that and the second click of a successful purchase gets told its hold expired, which is the API lying to a buyer who just bought.
- The hold response uses RETURNING on the acquire, so expiresAt is the database's value rather than the app server's arithmetic on the app server's clock.
- RETURNING also lets a partial multi-seat failure name WHICH seats were lost (requested ids minus the ids RETURNING gave back) instead of making the client refetch the whole map.
High-level design
10 min, end to end, no dives yet1) Users should be able to find events
- Search runs over Elasticsearch holding denormalized event documents: name, performer, venue, city, date, price range, all in one doc so a query is one hop.
- Event pages sit behind a CDN. The page for a hot event is requested millions of times and changes rarely, which is the definition of cacheable.
- Postgres holds the catalog as truth, and CDC (change data capture) streams changes into the index. The index is derived and rebuildable, so nothing catastrophic happens if it is seconds behind or has to be rebuilt from scratch.
- The point of this section is not that search is hard. It is that browsing is cacheable and boring, so it must be cleanly separated from the booking path.
- Every read served from the CDN or the index is a read that never touches the primary you are about to fight over.
2) Users should be able to see which seats are available
- The seat map cannot be perfectly live for 100K concurrent viewers.
- At 100K QPS against the booking primary, at exactly the moment it is committing sales, the read path would take down the write path.
- So a SeatMapService serves a cached snapshot of the event's seats, read from the seats table once per short TTL: one read per event per TTL, not one per viewer.
- Every reply carries the snapshot's asOf, so the client knows how old its picture is.
- The map is a hint, and the write at booking time is the referee.
- A stale map costs the user one retry and can never cost correctness, because nothing is decided by what the map said.
- Where it strains: everyone sees the same snapshot until the next refresh, and pushing every seat change to 100K viewers is the largest number in this design; the seat map freshness dive handles both.
3) Users should be able to hold seats and then buy them
- This is the heart of the design.
- BookingAPI owns the hold and the purchase, and both write the seats table in Postgres, the one record of who owns each seat.
- A hold sets each requested seat to HELD with a new hold_id and a hold_expires 10 minutes out: POST /v1/events/{id}/holds returns 201 { holdId, expiresAt }, or a 409 SEAT_UNAVAILABLE naming any seat that was not available.
- A purchase is one transaction: POST /v1/orders, minutes later, flips the held seats to SOLD and inserts the order row with its Idempotency-Key, so a sale is never half written. It returns 202 { orderId }.
- A ChargeWorker then charges the card, and the client polls GET /v1/orders/{id} for CONFIRMED or FAILED.
- The hold and the purchase are separate requests minutes apart, so the hold is data, a row with an expiry, not a lock or an open transaction. A buyer who walks away simply lets it expire.
- Cancel is DELETE /v1/holds/{holdId}, which sets the seats back to AVAILABLE.
- Where it strains: two buyers can reach for the same seat in the same millisecond, and a hold can lapse while its owner is still paying; the dive on selling a seat at most once handles both.
- And the charge after the commit can fail, or time out with no clear answer; the payment dive handles that.
4) The system should survive the on-sale spike
- Put a virtual waiting room in front of the event.
- Users get a queue position and a queue token before the sale opens. The admission token comes later, when the line reaches them (A4).
- An Admitter lets users in from the front of the line, and BookingAPI checks the admission token before it takes a hold.
- Where it strains: how fast the Admitter lets people in, and how much one admitted user may take, decide whether the booking path survives; the on-sale dive handles both.
- If they just nod and wait: "the riskiest part is the hold-then-confirm boundary, shall I open how a seat is never sold twice?"
- Running behind: protect the dives on selling a seat at most once and on the on-sale moment. The scale dive is the goalpost-move answer, do not open it unprompted.
Potential deep dives
~20 min, interviewer steers1) How is a seat never sold twice?
- The race is two buyers reaching for the same AVAILABLE seat in the same millisecond.
- Plus a second, sneakier race: your hold expiring while somebody else takes the seat, and you confirming anyway.
- There are three transitions, acquire, confirm and release, and all three are guarded UPDATEs. Compensation after a failed payment adds a fourth, guarded the same way.
- Run these statements at READ COMMITTED, because only there is "rows affected" the verdict on who won the seat.
- Read the map, see AVAILABLE, decide in application code that the seat is free, then UPDATE.
- The gap between the read and the write is the bug. Two users both read AVAILABLE, both decide yes, and both write. Nothing in the database ever objected.
- Adding a transaction around it does not help unless the write itself is conditional: the read told you a fact that was already stale by the time you acted on it.
- The gap between the read and the write is the bug. Two users both read AVAILABLE, both decide yes, and both write. Nothing in the database ever objected.
- SETNX on
seat:{event}:{seat}with a TTL, do the work, release. It works, and at small scale it works well.- It breaks on failover: a replica promoted mid-lease can hand the same lock to a second holder, and now two clients both believe they own the seat.
- The deeper point to make: even a correct lock still needs the resource to fence, because the lock holder can pause (GC, network) past its TTL and write late.
- If the seat row must check a token anyway, the lock is doing less than it appears to be doing.
- It also adds a second store that can disagree with the first, which is exactly the coupling you spend the rest of the design avoiding.
- The write itself decides who gets the seat: taking the hold (the acquire) is one conditional UPDATE. At READ COMMITTED, rows affected is the verdict: 1 means you won, 0 means somebody else did and you return 409.
- There is no window between deciding and writing because the decision IS the write.
- Name the isolation level, because at REPEATABLE READ or SERIALIZABLE the loser gets SQLSTATE 40001 instead of a zero-row result, which turns a clean 409 contract into 500s and a retry storm at exactly the wrong minute.
- Get the mechanism right, since interviewers push here. The loser did not "find" the new state: it blocked on the row lock held by the winner.
- When the winner committed, Postgres re-evaluated the WHERE against the newly committed version (EvalPlanQual) and skipped the row.
- That is why the count is trustworthy without you reading anything first.
- Expired holds are reclaimed inside that same acquire statement (
OR (status='HELD' AND hold_expires < clock_timestamp())), so no reaper sits on the WRITE path. - The confirm carries
AND hold_id=:my_hold, which answers "is this still MY hold" rather than "is somebody holding it". That one clause is what makes the deliberate transaction gap safe.- The SET list clears hold_id, so a SOLD row cannot be un-sold later by a release guarded on that hold. Every guard pins the state as well as the identity.
- The confirm commits the seat flip, the order insert and an outbox row (a note that the card still needs charging) together. Rows affected must equal the seat count or the transaction rolls back, so a partial multi-seat sale cannot exist, and a worker picks up the outbox row after the commit.
- The single-row compare-and-set (CAS) is the correctness mechanism, and any lock on top would only ever be an optimization.
- Locks reduce wasted work; they do not decide who owns the seat.
The statements, written out.
Acquire the hold.
UPDATE seats SET status='HELD', hold_id=:new_hold, order_id=NULL,
hold_expires = now() + interval '10 minutes'
WHERE event_id=:e AND seat_id=:s
AND (status='AVAILABLE'
OR (status='HELD' AND hold_expires < clock_timestamp()))
RETURNING seat_id, hold_expires;
-- rows affected = 1 means the seat is yours; 0 means 409 SEAT_UNAVAILABLE
Confirm the sale.
UPDATE seats SET status='SOLD', order_id=:o,
hold_id=NULL, hold_expires=NULL
WHERE event_id=:e AND seat_id=:s
AND status='HELD'
AND hold_id=:my_hold -- THE FENCE
AND hold_expires > clock_timestamp() - interval '5 seconds';
- Sketch this as SOLD.
- Once payments are on the table it writes PENDING_PAYMENT instead, and the payment worker flips it to SOLD on a definitive success.
- Why: SOLD otherwise means three different things at once.
- Once payments are on the table it writes PENDING_PAYMENT instead, and the payment worker flips it to SOLD on a definitive success.
Release on cancel. Write it out; this is where the bug lives if you only gesture at "the same shape":
UPDATE seats SET status='AVAILABLE', hold_id=NULL,
hold_expires=NULL, order_id=NULL
WHERE event_id=:e AND seat_id=:s
AND status='HELD' -- pins the STATE
AND hold_id=:h; -- pins the IDENTITY
now()istransaction_timestamp(), frozen at transaction start.- In a transaction that has been open a second or three,
hold_expires > now()silently grants a grace equal to however long you have been open. - And a multi-seat acquire under-grants the TTL by its own elapsed time.
- In a transaction that has been open a second or three,
- So: clock_timestamp() for the two comparisons, now() for SETTING hold_expires, which keeps every seat in one hold on one expiry.
- The confirm gets a small explicit grace (
> clock_timestamp() - interval '5 seconds') while the acquire reclaims at< clock_timestamp().- That overlap is deliberate, and the row lock referees it: whoever gets the row first wins, and the other statement sees the state the winner left.
- The confirm gets a small explicit grace (
- Confirming on
status='HELD'alone lets you sell a seat that a DIFFERENT customer now holds: your hold lapsed, theirs replaced it, the status is still HELD, and your UPDATE happily sells it out from under them.- Say the distinction in one line: status only answers "is somebody holding it". hold_id answers "is this still MY hold".
- The mirror-image bug is the expensive one. If the confirm does not clear hold_id, a SOLD row still carries a live hold_id.
- A release guarded only on hold_id then matches it and un-sells a paid seat: one row affected, one furious customer at the gate.
- A guard must pin the state as well as the identity.
- Same reason the acquire and the release both clear order_id: otherwise a FAILED order stays stamped on a seat the next customer is about to buy.
- The acquire and the confirm are NOT one transaction, and must not be.
- They are separated by minutes of user think-time while somebody types a card number.
- The strongest reason first: with pooled connections and stateless app servers, it is not expressible.
- The acquire and the confirm are separate HTTP requests, minutes apart, landing on different app servers holding different connections. There is no "the" transaction to keep open.
- And it would be an outage if you could: holding a row lock and a pooled connection across human time exhausts the pool, blocks every other buyer on that seat, and holds back vacuum.
- Vacuum specifically, because a transaction that has written keeps its xid live and the dead tuples behind it stay non-removable.
- Do not say it "pins a snapshot": at READ COMMITTED the snapshot is taken per statement and released, so the xid is the thing doing the damage.
- Vacuum specifically, because a transaction that has written keeps its xid live and the dead tuples behind it stay non-removable.
- So the hold is a lock converted into data: a row you can walk away from, visible to every server, expiring on its own without anyone holding anything.
- That deliberate gap is exactly why the fencing check is mandatory. You gave up the lock, so you must re-prove ownership at confirm time.
- Inside the confirm transaction the guarded seat flip goes FIRST, before the order insert and the outbox row. Putting the contended seat rows first is the right inter-table lock order: you take the hot lock, decide, and either bail immediately or hold it for the short remainder.
- Multi-seat, and the fix that actually works.
- Sorting your seat id array does nothing: lock order inside a multi-row UPDATE is the plan's row order, not your array order.
- So two customers can both send ascending lists and still deadlock when one gets an Index Scan and the other a Bitmap Heap Scan visiting the same rows in the opposite direction.
- Sorting your seat id array does nothing: lock order inside a multi-row UPDATE is the plan's row order, not your array order.
- Take the locks explicitly first, then run the guarded UPDATE:
SELECT seat_id FROM seats WHERE event_id=:e AND seat_id = ANY(:ids) ORDER BY seat_id FOR UPDATE; -- LockRows above the scan, order guaranteed -- then the guarded UPDATE over the same ids
- The alternative is one UPDATE per seat issued in sorted order, which gives the same guarantee at the cost of a round trip per seat.
- Either way, the confirm is also a multi-row UPDATE over the same rows and needs the same discipline, so it takes the same ordered lock step.
- Have a deadlock policy, since you raised deadlocks. On SQLSTATE 40P01 the whole transaction is aborted by Postgres.
- So retry it from the top, at most twice, with a few milliseconds of jitter; past that, return 409 and let the client pick again.
- A deadlock is never a partial write, so retry is always safe here.
- Dedupe seatIds server-side before the statement, or a duplicate id makes the rows-affected check fail spuriously.
- On a double-click, the seat guard makes the second click harmless: the seat is no longer HELD under that hold, so the second confirm matches zero rows. The Idempotency-Key then makes its RESPONSE correct.
- A hold expiring is the passage of time, not a write. No row changes, so there is no WAL record, no CDC delta, and no SSE push.
- The map shows HELD forever, and the lazy reclaim never fires because nobody reaches for a seat the map says is taken.
- Most on-sale holds are abandoned, so this is the common path, not an edge case.
- So keep the lazy reclaim for the write path, which is correct and keeps a reaper off the hot path.
- And add a sweeper for the READ path: it expires abandoned holds so a real UPDATE happens and a delta is emitted.
- Say the asymmetry plainly: the write path tolerates a dead sweeper, the read path does not. If the sweeper dies, sales stay correct and the map goes stale, which is the right way round.
- Anything reading availability has to apply the same predicate, including the degraded view: a section count is
count(*) WHERE status='AVAILABLE' OR (status='HELD' AND hold_expires < now()), notstatus='AVAILABLE'.- Otherwise the page under-reports free seats by however far behind the sweeper is.
2) What happens at the on-sale moment?
- The load is not just large, it is concentrated: one event, one shard, one set of 50K rows, one minute.
- Read replicas do not help, because this is a write contention problem, and the writes are all on one event.
- Sharding does not help either, and say why rather than sloganeering: seats within an event are trivially shardable, so the hot event is one partition only because event_id is the shard key.
- And event_id is the shard key because a multi-seat purchase has to be one transaction and therefore one shard.
- Sharding contains the blast radius; it does not divide the contention.
- What actually happens: the connection pool exhausts, p99 goes vertical, most requests time out, and every timeout becomes a retry, so offered load rises exactly when capacity falls.
- Cap requests per second at the edge and 429 the rest. It does protect the database, which is why it is better than nothing.
- But it sheds at random, so a user who arrived first has no better chance than one who arrived last.
- That is not a fair sale, and fans experience it as a lottery run by your infrastructure.
- It also invites a retry storm, because a 429 tells the client to try again immediately, and the client that retries hardest wins. That client is usually a bot.
- But it sheds at random, so a user who arrived first has no better chance than one who arrived last.
- Users enter the room before the sale opens and receive a position and a queue token, so the crowd is measured and ordered before it is served.
- The wait becomes explicit, ordered, and honest: you know you are #412,908 and roughly what that means.
- Admission is a concurrency limiter: it keeps at most N sessions in flight in the protected zone (everything behind the queue: the booking path, its connection pool, and the primary it writes to). It is a bulkhead.
- N comes from pool size and measured p99 by Little's law, seeded by a pre-sale capacity test, with commit rate and 409 rate as feedback that adjusts N.
- A token bucket refilled by commit rate cannot bootstrap, because at t=0 the commit rate is zero: nobody has been admitted yet.
- When Postgres slows, p99 rises, N falls, and the queue moves slower; nothing fails.
- The room is the backpressure: it converts a 1M-request thundering herd into a steady stream the protected zone can serve, which is why the connection pool and the seat map cache survive.
- The connection pool, the seat map cache, and the booking primary all see a load you chose rather than a load that arrived.
- Even a perfect design loses the sale to denial of inventory: admitted bots hold every seat and never pay.
- Nothing above caps how many seats one admitted session can hold, so a bot farm admitted in perfect order holds the whole stadium in seconds while every CAS legitimately returns one row and the map correctly reports sold out.
- The fix is three caps: hold cap per identity at about 8, TTL cut to about 2 minutes during the on-sale (back to 10 afterwards), and best-available assignment instead of letting people pick freely on hot sales.
- The position counter is itself a hot key, and a worse one than booking.
- A million ordered positions in sixty seconds is about 17K writes/sec against one per-event structure, and the ordering is exactly what you cannot split across shards.
- So do not promise exact ordering: either positions are approximate, or each app server draws blocks of positions and hands them out locally, which keeps ordering coarse but the writes rare.
- And say what happens if that store dies mid-sale: admission falls back to a fixed conservative N with no position display, because a missing wait estimate is survivable and an unpaced crowd is not.
- Be honest that fairness is a product decision: first-come order is one policy, a lottery among everyone who joined in a window is another, and the business picks.
- If bots are in scope, the lottery is the strong answer precisely because it makes speed stop mattering: bind one position per verified account per event, require payment history, challenge at join.
- If bots are out of scope, cut them cleanly and stop using them as the argument for the queue.
3) How fresh is the seat map, and what does staleness cost?
- 100K viewers want a picture of state that changes about 330 times a second.
- Perfect freshness is not merely expensive here, it is the thing that would break the write path.
- 100K QPS of seat-map reads against the booking primary, at exactly the moment it is committing sales.
- Even if the reads themselves are cheap, they consume the connections and buffer cache the writes need, so the correctness-critical path starves to serve a picture.
- One read per event per TTL instead of one per viewer. This is most of the win and should be said first.
- The residual problem is uniformity: everyone sees the SAME stale snapshot, so everyone reaches for the same seats that looked free in it, and the 409 rate spikes in bursts synchronized to the TTL.
- It also gives the client no way to know how stale its picture is, so the UI cannot be honest about it.
- Serve the snapshot from cache, then stream deltas over SSE fed by CDC on the seat table, so the picture converges continuously instead of stepping once per TTL.
- Expose asOf in the payload. The client can gray out contested seats, show "updated 0.4s ago", and set expectations before the user clicks.
- The fan-out decides the granularity: section-level counts by default, per-seat detail only for the section in view. About 330 seat transitions per second times 100K subscribers is roughly 33M messages per second and order of 1 GB/s of egress for one show. So:
- section-level counts ("Section 112: 43 seats left") are the default view
- per-seat detail is streamed only for the client's current section or viewport
- deltas are coalesced on a one-second tick
- only AVAILABLE to SOLD transitions are published
- Suppressing HELD churn removes most of the volume and stops the stream from being a live inventory feed for scrapers.
- Watermark the stream: the snapshot carries the CDC LSN (log sequence number, Postgres's position in its write-ahead log) it was built at, and deltas carry monotonic sequence numbers.
- Clients drop anything at or below their watermark and resnapshot only on a detected gap.
- Reconnects use jittered backoff against a single-flight snapshot endpoint.
- Without that a client silently misses a delta, and "just re-GET the snapshot on reconnect" is a 1M-client stampede at the worst moment of the day.
- An expired hold writes nothing, so CDC emits no delta for it. The read-path sweeper exists to turn abandoned holds into real writes.
- Every availability count also has to read
AVAILABLE OR (HELD AND hold_expires < now()), or the map under-reports free seats. - Close with the framing: the map is a hint and the CAS is the referee, so a stale map costs one retry and never costs correctness.
- That sentence is what makes all the caching above defensible instead of sloppy.
4) What if payment fails or the processor times out ambiguously?
- The sell transaction commits, then a worker calls the processor, and a timeout is read as a success or a failure.
- A timeout after sending means the charge may or may not have happened. Retry it with no key and the card can be charged twice; call it failed and release the seat, and a paid seat goes back on sale.
- A worker that dies between the commit and the charge leaves a sold seat that is never charged, because nothing records that the charge is owed.
- The orderId is the processor's idempotency key on every attempt, so a retry can never become a second charge.
- The transaction that sold the seat also writes an outbox row, so a worker dying between commit and charge leaves work that a sweeper picks up, not work that is lost.
- What is still wrong: SOLD means three different things, paid, charge in flight, and charge ambiguous, and a definitive failure has no clean way back.
- Releasing the seat with the hold_id guard matches zero rows, because the seat is no longer HELD and its hold lapsed long ago.
- The charge is keyed by orderId, presented as the processor's idempotency key on every attempt, so a retry across crashes, restarts, and sweeps can never become a second charge.
- The seat is already committed before the money exists, so say that uncomfortable part explicitly.
- That is the price of not holding a lock across the processor call, and it makes compensation a designed path rather than an assumption.
- Never guess on a timeout. A timeout after sending means the charge may or may not have happened.
- There are exactly two honest moves: retry with the same key, or query the processor's status endpoint for that key.
- Only a definitive answer moves the order to CONFIRMED or FAILED.
- On a definitive failure, compensate with a guard keyed by the order, not by hold_id. Reusing the hold_id guard matches zero rows, because the seat is no longer HELD and its hold lapsed long ago.
- Give the seat a PENDING_PAYMENT state, because SOLD currently means three different things: paid, charge in flight, and charge ambiguous.
- The seat flips to PENDING_PAYMENT at commit and to SOLD on a definitive success.
- order.status is the money truth and the gate scanner trusts it, not the seat row.
- Bound how long a seat may sit unpaid during ambiguity (minutes, not hours), then force a resolution rather than leaving it parked.
- The outbox sweeper finishes the job after a crash. The transaction that sold the seat also wrote the outbox row, so a worker dying between commit and charge leaves work that is picked up, not work that is lost.
UPDATE seats SET status='AVAILABLE', order_id=NULL,
hold_id=NULL, hold_expires=NULL
WHERE event_id=:e AND order_id=:o AND status IN ('SOLD','PENDING_PAYMENT');
-- then mark the order FAILED and notify
- The good news worth saying: that guard is safe against a concurrent re-hold and is idempotent.
- No acquire predicate matches a SOLD or PENDING_PAYMENT row, so nobody can be mid-acquire on it, and a second run of the compensation matches zero rows and does nothing.
- Clearing order_id matters too, or a FAILED order stays stamped on a seat the next buyer is about to take.
- The seat map handles a sold-then-released seat the way it handles everything else, as a delta.
- Name the genuinely nasty case rather than hiding it: the processor settles the original charge after you declared failure, released the seat, and resold it.
- The second sale is real and cannot be un-resold, so this resolves as money, not as inventory: auto-refund the first charge, detect it from the nightly settlement reconciliation, and issue a make-good.
- Bounding the ambiguity window is what keeps this rare.
5) Scale, and what melts first
- Seats inside one event are trivially shardable, so splitting a hot event across shards looks like it divides the contention.
- But a multi-seat purchase then spans shards, so the seat flip and the order insert stop being one transaction, and an all-or-nothing sale needs a distributed transaction.
- And it attacks the wrong number: the database is not what melts first.
- All the seats for one event live together, so a multi-seat purchase stays one transaction on one shard, and a hot event's contention stays inside that shard.
- What is still wrong: it treats the database as the bottleneck, when 50K seats over a few minutes is a few hundred commits per second that one primary handles.
- The seat map read path and the connection pool melt first, and an async standby promoted on failover can bring back seats already sold.
- What melts first is the seat map read path and its fan-out, not the database.
- 100K QPS of map requests arrives before the sale even opens, while the write rate is still zero.
- Once seats start moving, 330 deltas per second to 100K subscribers is 33M messages per second if you stream per-seat to everyone.
- That number is why section counts are the default and per-seat detail is scoped to the section in view.
- The second thing to melt is the connection pool, because every admitted user holds a connection for the length of their request and a spike turns queueing into timeouts, which turn into retries.
- That is exactly what the waiting room protects, and it is worth saying in those terms: the queue exists because the two things that break first are both a function of concurrent users, not of seats sold.
- Shard by event_id. All the seats for one event live together, so the guarded CAS and the multi-seat transaction stay single-shard, and a hot event's contention is contained inside one shard instead of smeared across all of them.
- Synchronous commit to at least one standby on the seats shards, so a failover loses no committed sale (RPO=0, a recovery point objective of zero). The whole correctness story is one Postgres row.
- With async replication a failover promotes a standby that is missing committed sales: seats resurrect as AVAILABLE and get sold a second time, which is the one NFR that must never break.
- The extra commit latency is affordable precisely because this design was built not to care about write throughput.
- Instrument the invariant, not just the boxes. NFR one is "a seat is sold at most once", so:
- run a continuous query that alarms on any seat with more than one confirmed order
- reconcile nightly against the processor's settlement file in BOTH directions (charges with no order, orders with no charge)
- page on outbox depth and oldest-unclaimed-age as the payment SLO
- track 409 rate and hold-to-purchase conversion as the fairness signals
- A conversion rate falling toward zero while holds climb is denial of inventory in progress.
- Browsing is served from the CDN and Elasticsearch, never from the booking primary. Catalog reads go to read replicas. The hot event's seat map is served from cache.
- Sizing sanity check: 50K seats sold over a few minutes is only a few hundred commits per second on one shard, which a single primary handles comfortably.
- The design is not fighting write volume, it is fighting arrival concurrency.
- State the contention story precisely, because the averages hide it. Write contention is spread across 50K rows at a few hundred commits per second, which is nothing.
- But popularity inside one event is heavily skewed, so front-section rows take thousands of failed CASes and near-100% 409 rates while the upper deck sees essentially none.
- The hot key is a few hundred seats inside a hot event, not the event as a whole, and that is what the queue and best-available assignment are really smoothing.
- General admission is the real single hot key, and it is the one case where "a single hot key" is literally true.
- GA has no seat rows, just a quantity, so the CAS collapses to one counter row that every buyer serializes on.
- Fix it by bucketing: split the inventory into K counter rows, hash each buyer to a bucket and fall back to scanning the others when theirs is empty.
- Reconcile the last few units under a single-bucket fallback, where contention is trivially low because almost nothing is left.
- Both clients POST /holds with the same seatId, and both requests run the same conditional UPDATE: SET status='HELD', hold_id=:new_hold, order_id=NULL, hold_expires=now()+10min WHERE event_id=:e AND seat_id=:s AND (status='AVAILABLE' OR (status='HELD' AND hold_expires < clock_timestamp())) RETURNING hold_expires.
- We are at READ COMMITTED, which is what makes rows-affected a verdict at all.
- Postgres serializes them on the row: the first commits, flips the row, and gets rows affected = 1, so it returns 201 with a holdId and the database's own expiry.
- The second did not "find" the new state, it blocked on the row lock.
- When the winner committed, Postgres re-evaluated the WHERE against the newly committed version (EvalPlanQual) and skipped the row, so rows affected = 0 and it returns 409 SEAT_UNAVAILABLE.
- Worth naming the isolation level explicitly: at REPEATABLE READ or SERIALIZABLE the loser would get SQLSTATE 40001 instead, turning a clean 409 into a 500 and a retry storm.
- There is no gap between deciding and writing because the decision IS the write, and no application code ever compares a read to a state it hopes is still true.
- The loser's client refetches the seat map (which by then has the delta over SSE) and picks another seat.
- Note what did NOT happen: no lock service, no coordination, no distributed transaction. The row is the referee.
- They see a 409 HOLD_EXPIRED and their card is not charged.
- Mechanically: the confirm is an UPDATE ... SET status='SOLD', order_id=:o, hold_id=NULL, hold_expires=NULL WHERE ... AND status='HELD' AND hold_id=:my_hold AND hold_expires > clock_timestamp() - interval '5 seconds'.
- If their hold lapsed and another buyer re-acquired the seat, the row now carries a different hold_id (and the acquire statement reclaimed it in the same breath).
- So the guard matches zero rows, the whole transaction rolls back, and no order, no outbox row, and no charge exist.
- Two details that are easy to drop and both bite.
- One: the SET list must CLEAR hold_id, or a SOLD row keeps a live hold_id and a release guarded only on hold_id will un-sell a paid seat, which is why every guard pins the state as well as the identity.
- Two: the comparison uses clock_timestamp(), because now() is transaction_timestamp() and would silently grant a grace equal to however long the transaction has been open.
- This is also why hold_id is a fencing token rather than decoration: confirming on status='HELD' alone would have sold the seat that a DIFFERENT customer now holds.
- Status only answers "is somebody holding it" while hold_id answers "is this still MY hold".
- The gap is deliberate, and with pooled connections and stateless app servers it is not even optional: the acquire and the confirm are separate HTTP requests landing on different servers and different connections, so there is no single transaction to hold open.
- And if there were, it would pin a live xid, hold back vacuum, and block every other buyer on that seat.
- The hold is a lock converted into data, and the fence is the price of that conversion.
- Not the database. The first thing to break is the fan-out on the read path, and it is worth computing out loud.
- 50K seats transitioning across the sale window is about 330 deltas per second, and 100K subscribers on one event makes that roughly 33M messages per second and order of 1 GB/s of egress for one show.
- So section counts are the default, per-seat detail is scoped to the client's current section, deltas are coalesced on a one-second tick, and only AVAILABLE to SOLD transitions are published.
- Even before the sale opens, 100K QPS of map requests arrives while the write rate is still zero, so those reads are served from a cached snapshot with an asOf, never from the primary.
- 50K seats transitioning across the sale window is about 330 deltas per second, and 100K subscribers on one event makes that roughly 33M messages per second and order of 1 GB/s of egress for one show.
- The second thing to break is the connection pool, because every admitted user holds a connection for the length of their request.
- Queueing turns into timeouts and every timeout becomes a retry, so offered load rises as capacity falls.
- Both failures are functions of concurrent users, not of seats sold, which is exactly what the waiting room addresses.
- And it does it as a CONCURRENCY limiter: hold N sessions in flight in the protected zone, N derived from pool size and measured p99 by Little's law and adjusted by commit rate and 409 rate.
- Pacing on commit rate alone cannot bootstrap, since at t=0 nobody has been admitted and the commit rate is zero.
- And it does it as a CONCURRENCY limiter: hold N sessions in flight in the protected zone, N derived from pool size and measured p99 by Little's law and adjusted by commit rate and 409 rate.
- The database itself is comparatively fine: 50K seats over a few minutes is a few hundred commits per second on one shard, though popularity skew means the good rows still see near-100% 409 rates.
- And handle the shard question precisely: seats within an event are trivially shardable, so the hot event is one partition only because event_id is the shard key, and event_id is the shard key because a multi-seat purchase must stay one transaction and therefore one shard.
- Never guess. The order stays in its pending state and we do exactly one of two things: retry with the same key, or query the processor's status endpoint for that key.
- The key is orderId, so a retry can never become a second charge, and only a definitive answer moves the order to CONFIRMED or FAILED.
- If the worker crashed instead of timing out, the outbox row written inside the sell transaction is still there, and the sweeper picks it up and finishes charge-and-deliver.
- On a definitive failure we compensate, and the guard has to change: releasing on hold_id would match zero rows, because the seat is no longer HELD and its hold lapsed long ago.
- So the statement is UPDATE seats SET status='AVAILABLE', order_id=NULL, hold_id=NULL, hold_expires=NULL WHERE event_id=:e AND order_id=:o AND status IN ('SOLD','PENDING_PAYMENT'), then mark the order FAILED.
- That guard is safe against a concurrent re-hold and idempotent, since no acquire predicate matches a sold row and a second run affects nothing.
- The uncomfortable part to say out loud is that the seat is already committed before the money exists, because we deliberately do not hold anything across the processor call.
- Which is also why the seat sits in PENDING_PAYMENT while the charge is in flight, and order.status stays the money truth the gate scanner trusts.
- So compensation is a designed path, not an assumption, and the seat map treats a sold-then-released seat as just another delta.
- The genuinely nasty case is the processor settling the original charge AFTER we declared failure and resold the seat.
- That one cannot be fixed in inventory, so it resolves as money: an auto-refund caught by the nightly two-way settlement reconciliation plus a make-good.
- Because it would not be doing the job it looks like it is doing.
- A Redis lock per seat works until failover, when a promoted replica can hand the same lock to a second holder.
- And even a perfectly correct lock still needs the resource to fence, since a holder can pause past its TTL and write late.
- If the seat row has to check a token anyway, the lock is an optimization, not the correctness mechanism.
- Meanwhile the seat row already gives us a conditional UPDATE whose rows-affected count is an atomic verdict, in the same store and the same transaction as the order insert and the outbox row, so the sale is all-or-nothing without a distributed transaction.
- Adding Redis would add a second store that can disagree with the first, and reconciling a lock state with a row state is the coupling this whole design exists to avoid.
- Redis is a fine place for the waiting room queue and the seat map cache, which are both things that can be rebuilt.
- It is the wrong place for the fact that decides who owns a seat.
Final design + what is expected at each level
wrapFinal Design
- Browsing is cached and derived: CDN in front of event pages, Elasticsearch for search, catalog truth in Postgres with CDC feeding the index.
- The seat map is a cached snapshot carrying an asOf and the CDC LSN it was built at, kept moving by sequence-numbered SSE deltas, section counts by default and per-seat detail only for the section in view.
- The waiting room holds ordered positions and admits by concurrency, keeping at most N sessions in flight on the booking path behind it, which makes it backpressure rather than a UI feature. Per-identity hold caps and a short on-sale TTL stop an admitted bot farm from holding the stadium.
- The booking path is two transactions on purpose. First a guarded acquire that also reclaims expired holds on the write path.
- Then a guarded confirm fenced by hold_id that flips the seats, clears hold_id, inserts the order, and writes the outbox row in one commit, seats first.
- A read-path sweeper expires abandoned holds so the map gets a delta, since an expiry by itself writes nothing. The write path survives a dead sweeper; the read path does not.
- The charge happens after the commit, keyed by orderId, with the seat in PENDING_PAYMENT until the answer is definitive, a sweeper for crashes, and a compensation path guarded on order_id.
- Seats are sharded by event_id, so a multi-seat purchase stays one transaction on one shard, and the seats shards commit synchronously to a standby so a failover cannot resurrect sold seats.
- The invariant is monitored, not assumed: a continuous query alarms on any seat with more than one confirmed order, and settlement reconciles both ways nightly.
What is Expected at Each Level
- Mid-level candidates are expected to produce the seat, hold and order model, and to know that the write must be guarded rather than decided in application code after a read.
- Senior candidates are expected to:
- nail the hold_id fencing token AND clear it on the confirm
- guard every transition on state as well as identity
- name the isolation level that makes rows-affected a verdict
- state the transaction boundary explicitly (with pooled connections it is not even expressible, and it would hold back vacuum if it were)
- use an outbox so the charge happens after the commit, at-most-once in effect (the outbox delivers at least once, the idempotency key at the processor collapses the retries)
- on the on-sale moment, go past scaling the database to admission at the edge, and say why rate limiting that sheds at random is unfair and invites a retry storm
- on seat map freshness, serve the map from a short-TTL cache, one read per event instead of one per viewer, and say that everyone then sees the same stale snapshot
- on scale, shard by event_id so a multi-seat purchase stays one transaction on one shard
- Staff candidates are expected to:
- open with the concentration math that reframes the problem from throughput to skewed contention inside one hot event
- compute the SSE fan-out and let that number pick the seat map design rather than noting it as an aside
- treat the waiting room as a concurrency limiter rather than a UI feature, and see the denial of inventory it does not prevent
- monitor the at-most-once invariant instead of asserting it
- name honestly what the design does not solve: the processor that settles a charge after the seat was resold, and bots as the actual adversary behind every fairness mechanism
Write the acquire, the confirm, and the release from memory on paper: the OR-reclaim clause, the hold_id fence, the SET that clears hold_id, and the release guarded on status AND hold_id.
Then say the fan-out number out loud (330 deltas/s times 100K subscribers is about 33M messages/s).
Then read the Final Design summary aloud once, timing yourself; if it runs past 90 seconds, cut words until it fits.
How the Admitter actually processes the queue
reference, not one of the 5 stepsA1) The mechanism
- The Admitter, the waiting room's admission loop, does not walk the queue: it advances one number per event, admitted_through.
admitted_throughis a single integer per event, and everyone whose position is at or below it may enter.- Admitting 5,000 fans is therefore one INCRBY, not 5,000 messages.
- Clients hold an SSE stream or poll, and ask one question: has the line passed my position yet.
- The alternative is popping a million entries and notifying each one, which rebuilds the thundering herd inside the thing that exists to prevent it.
- The fans hold still and the line sweeps past them.
- Step 3 of the diagram draws the crowd shuffling forward, which is the right mental model to say out loud; the implementation inverts it.
- The Admitter runs one tick every 500ms, and the tick has to be one round trip, for a reason in A5.
local in_flight = redis.call('ZCOUNT', inflight, now, '+inf') -- the lease set, see A3
local slots = N - in_flight -- how much room the zone has
if slots <= 0 then return 0 end
local adv = math.floor(slots / claim_rate) -- see A2
return redis.call('INCRBY', admitted_through, adv) -- the whole admission
A2) Why the division by claim_rate
- A large share of queue positions are dead: closed tabs, people who gave up, duplicate joins.
- Advance by exactly
slotsand a 40% no-show rate runs the protected zone at 60% of N, so the sale takes almost twice as long for no reason.
- Advance by exactly
- So measure the share of positions that show up, claim_rate, rather than guessing it.
- Of the positions the line passed in the last 30 seconds, what share actually exchanged a token for a session inside the grace window (defined two bullets down).
- That is
claim_rate, and dividing by it is a feedback loop that self-corrects when the crowd's behaviour changes mid-sale. - Too high starves the zone, too low overfills it, which is the same failure as getting N wrong.
- The grace window is the other half: an admitted fan has about 60 seconds to show up.
- Once the line passes you, you have about 60 seconds to actually show up and trade the position for a session.
- Miss it and the admission is void and you rejoin at the back, because without expiry a ghost holds capacity forever.
A3) How a slot ever frees, which is the part usually gotten wrong
- A session ends three ways, and only one of them can tell you about itself.
- Clean exit: checkout finishes or the user leaves, so an explicit release is possible.
- Abandoned tab: nothing is sent, ever.
- Your own server dies while holding the count.
- So the in-flight count cannot be a counter you increment and decrement: each admitted session is a lease that expires.
- One sorted set per event, scored by when that session's lease expires:
- admit is
ZADD inflight <now+120s> <sessionId> - heartbeat is the same ZADD with a fresh score, sent by the live tab
- in_flight is
ZCOUNT inflight <now> +inf, one cheap read - the sweep is
ZREMRANGEBYSCORE inflight -inf <now>
- admit is
- A dead tab stops heartbeating, its lease expires, and the slot returns with nobody telling you.
- ZADD is an upsert, which is the only reason the heartbeat can be the same command.
- A sorted set holds unique members, so an existing member cannot be added twice.
- Member already there: its score is overwritten in place.
- Member not there: it is added.
- The reply counts NEW members only, so a successful heartbeat replies 0.
- One sorted set per event, scored by when that session's lease expires:
- Each session is a member of one Redis sorted set scored by its lease expiry, so the sweep frees every slot whose score is below now. The walkthrough starts with three sessions, member on the left and expiry score on the right:
before sess_A 120
sess_B 125
sess_C 130
run ZADD inflight 180 sess_A sess_A is already a member
reply 0 = zero NEW members added
after sess_B 125
sess_C 130
sess_A 180 score 120 -> 180, so it re-sorted to the end
still 3 members, no duplicate
before sess_A 120
sess_B 125
sess_C 130
run ZADD inflight 200 sess_D sess_D is not a member yet
reply 1 = one NEW member added
after sess_A 120
sess_B 125
sess_C 130
sess_D 200 appended, now 4 members
- Say out loud that this is the seat hold one level up.
- Same expiry-and-reclaim shape as
hold_expireson the seat row, applied to admission instead of inventory. - Two instances of one pattern reads far better than two unrelated tricks.
- Same expiry-and-reclaim shape as
A4) Who calls what, and where the token comes from
- Redis is never reachable from the browser. Every command in A3 is server side.
- BookingAPI, the service behind the queue, writes the leases: it makes both ZADD calls.
- On admission: the browser POSTs
/enterwith the token, BookingAPI verifies it, then writes the lease. - On heartbeat: the browser POSTs
/heartbeatevery 30s, BookingAPI writes the same lease with a later score. - The browser's whole job is to keep pinging. It never speaks to Redis.
- On admission: the browser POSTs
- The Admitter reads and sweeps the leases and never writes one.
- ZCOUNT on each 500ms tick, which is where in_flight comes from.
- ZREMRANGEBYSCORE on the same tick, housekeeping only.
- The one thing it writes is
admitted_through.
- There are two tokens, easy to conflate: the queue token proves your position, the admission token proves you may enter now.
- Queue token: handed out at join time, before the sale. It proves "I hold position 412,908".
- Admission token: handed out when the line reaches that position. It proves "I may enter now".
- WaitingRoom signs the admission token and pushes it over SSE once the line passes the fan's position:
- joins before the sale, and WaitingRoom returns a position plus a queue token
- the browser holds an SSE stream asking whether the line has reached that position
- the Admitter advances
admitted_throughpast it - WaitingRoom signs an admission token and sends it down that stream
- the browser POSTs
/enterwith it, and BookingAPI writes the first ZADD
- The admission token carries event id, position, session id, and an expiry of about 60s, which IS the grace window from A2.
- It is bound to one account, so it cannot be resold or handed around a bot farm.
- HMAC signed by WaitingRoom and verified at the edge against a shared secret, so there is no lookup per request.
A5) Two admitters must not both see the same free capacity
- If two replicas each read
in_flight = 4,200againstN = 5,000, they both admit 800 and the zone overshoots by 800 sessions. - Do the read and the advance in one Lua script so the count and the INCRBY cannot interleave.
- That beats electing a leader per event: no election to run, and an admitter dying mid-tick costs one tick instead of a failover.
- Same instinct as the signed token in A4: keep the per-event hot key from coming back, whether the pressure is one tick or one request.
A6) The failure nobody notices, and what the fan sees
- If the tick loop stops, nothing errors.
- Postgres is healthy, the pool is empty, every dashboard is green, and the sale is dead because nobody is being let in.
- So alarm on the invariant: positions admitted in the last 30 seconds is zero while queue depth is above zero. Zero is the alarm, not the quiet.
- If Redis dies, admission falls back to a fixed conservative in-flight limit N with no position display, because a missing wait estimate is survivable and an unpaced crowd is not.
- The ETA is
(my_position - admitted_through) / admission rate, where the rate is an EWMA (exponentially weighted moving average) over the last 30 seconds.- Label it approximate, and never let it jump upward on screen if you can avoid it, because an ETA that grows is a support ticket.
- The position counter is itself the hot key from the on-sale dive, so blocks of positions are drawn per app server and exact ordering is not promised.