Design a trading system that prevents fraud

Functional Requirements

1. Every order should pass a risk decision before it is routed, inside the trading latency budget, with the mandatory pre-trade controls applied without exception.
2. The system should detect fraudulent patterns across accounts, sessions and devices in near real time, and be able to freeze an account.
3. Withdrawals and other irreversible egress should be gated on a stronger, slower check than orders are.
4. Every decision should be explainable and auditable years later, and an analyst should be able to review and reverse it.

Below the line (out of scope):

• Identity verification at onboarding. It is a vendor call and a different problem: I am defending accounts that already exist and have already passed KYC.
• The matching engine, market data, and order routing. Say the cut out loud, because it is what lets me treat "allowed" as the end of my responsibility.
• Margin and credit risk. Those are market risk, not fraud, and conflating them is a common way to lose half the hour.
• Tax reporting and statements, and the analyst tooling's own UI.

Non-Functional Requirements

• Do the two-rate comparison first, because it decides where the expensive checks are allowed to live.
  • 50M orders a day is about 580 a second averaged over 24 hours, but the market is open 6.5 hours, so the in-session average is about 2,100 a second, and the open concentrates it further: call it 5,000 a second at peak.
  • 200K withdrawals a day is 2.3 a second. Flat, all day, no open-bell spike.
  • That is 250 to 1 by daily count, and far wider at peak. Anything I want to spend 500 ms on is affordable on the withdrawal path and unthinkable on the order path.
  • So the design is not "how do I make fraud checks fast". It is "which checks have to be on the fast path at all", and the answer is: almost none.
• Then the loss asymmetry, which is the other half of the same idea. Say it as bounded versus total, not as reversible versus irreversible, because a filled trade is not reversible.
  • A bad order costs market exposure. An unfilled one can be cancelled; a filled one generally cannot be undone, since clearly-erroneous busts are a narrow venue remedy for off-market prices and fraud is not a ground for one.
  • What makes it survivable is that the loss is BOUNDED, by the notional cap times the move over the seconds until detection, and the position is still an asset that can be liquidated.
  • A completed withdrawal is gone. There is no recall on a wire, and an ACH pull can be returned against me for up to 60 days after I have already paid out.
  • So the two paths get opposite failure policies, which is the single most useful sentence on this page and the thing I would lead the dives with.
• Latency: p99 under 100 ms for an order end to end, and the risk decision gets 10 ms of it. That is a hard budget handed to me, not one I get to negotiate, because the rest belongs to routing and the venue.
• Say which entity you are in one sentence, because the obligations differ: I am the retail broker, the matching venue is downstream and not mine. That decides who owns 15c3-5, who can bust a trade (the venue, and not for fraud), and where self-trade prevention actually executes.
• Availability: the risk system must not be able to stop customers trading. A broker that stops accepting orders because a fraud service is down has caused a larger incident than the fraud would have, and has handed an attacker a denial-of-service lever.
• Explainability is a functional requirement wearing a non-functional hat. Every decision stores its inputs, the rule version and the score version. A regulator will ask why this account was frozen, and "the model said so" is not an answer anyone accepts.
• Retention: 7 years, immutable. SEC 17a-4 is six years for the core books and records and the BSA is five for SAR-related material, so seven is a covering envelope rather than a magic number. At roughly 1 KB a decision that is 50 GB a day and about 130 TB, which is cheap.
• Feedback latency: a confirmed fraud case must change behaviour in minutes. If the loop from "analyst confirms" to "the inline path knows" is a nightly model retrain, the same attacker runs all night.
• False positives have a real, asymmetric cost. Blocking a legitimate trade during a market move loses the customer money and generates a complaint I will have to answer in writing. This is not a system where you turn the threshold up and call it safety.

Defining the Core Entities

• Account is the customer relationship: id, status (active, restricted, frozen), tier, and the KYC record it was opened against.
• Session and Device are separate on purpose.
  • A device fingerprint outlives a session and is what links accounts an attacker is running together.
  • Binding a session to a device is what makes "this order came from somewhere new" a signal rather than a guess.
• Order is an intent to trade: account, instrument, side, quantity, and the session and device it came from. Carrying the device on the order is what lets the stream correlate later without a join.
• Transfer covers deposits and withdrawals, and it carries the thing that matters most here: settledAt. A deposit that has landed is not a deposit that is final.
• Beneficiary is a withdrawal destination: bank account or wallet, with addedAt. The age of this row is one of the strongest signals in the system.
• RiskDecision is immutable and is the audit trail: the verdict, the features it saw, the rule version, the score version, and the latency it took.
  • It records the INPUTS, not just the answer. Re-deriving why a decision was made two years later is impossible otherwise, because the score has moved a thousand times since.
• Case is the analyst's unit of work: the accounts involved, the evidence, the state, and the outcome. A closed case is also a training label, which is the only place clean labels come from.
• Rule is versioned and deployable on its own, with a shadow mode. Fraud moves faster than a release train, so rules cannot be code.

API or System Interface

POST   /v1/orders                        // the hot path, 10 ms of risk budget
                                         // → 201 { orderId } | 202 { stepUpRequired } | 403 { reason, decisionId }

POST   /v1/transfers/withdrawals         // { amount, beneficiaryId }
                                         // → 202 { transferId, status: HELD | RELEASED, eligibleAt }
POST   /v1/beneficiaries                 // → 201 { beneficiaryId, withdrawableAt }  // starts the cool-off

POST   /v1/accounts/{id}/freeze          // internal, and reversible on purpose
GET    /v1/cases?state=open&sort=exposure
POST   /v1/cases/{id}/resolve            // { outcome, notes }  -> becomes a training label
POST   /v1/rules                         // { definition, mode: shadow | enforcing }

• The order endpoint has three outcomes, not two, and the middle one is the useful one.
  • Allow and block are the obvious pair. Step-up (re-authenticate, then retry) is what lets the system act on suspicion without eating a false positive.
  • Most real signals are ambiguous. A design with only allow and block forces every ambiguous case into one of two wrong answers.
• Withdrawal returns 202 with a status, always. There is no synchronous success. Making the held state normal rather than exceptional is what stops "held" from reading as an outage to the client.
• Adding a beneficiary returns withdrawableAt in the response. The cool-off is visible at the moment it starts, not discovered later when a withdrawal is refused, which is the difference between a considered policy and a bug.
• decisionId comes back on a block. Support can look up exactly what happened without an engineer, and the customer gets an answer.
• Rules deploy in shadow mode first. A new rule runs and logs its verdict without enforcing it, so its false-positive rate is measured on real traffic before it can refuse anybody.
• What the API deliberately does not say: a user under investigation is never told. Tipping off is itself an offence in most jurisdictions, so the product surface for a SAR-flagged account looks like an ordinary hold.

1) A risk decision on every order, inside the budget

• The inline path computes almost nothing, and that is the design rather than a compromise.
  • It reads one precomputed score for the account out of a feature store, about 2 ms.
  • It runs deterministic rules that need no joins: velocity caps, watchlist membership, self-trade prevention, notional limits. About 1 ms.
  • It returns allow, step-up, or block, and writes the decision asynchronously.
• No model runs here, and be precise about why, because the sloppy version of this claim is easy to puncture.
  • It is not that inference is slow. A compiled gradient-boosted tree scores a prepared feature vector in tens of microseconds in-process, and card networks score every authorisation inline.
  • What is unaffordable is computing the FEATURES, which needs joins over history, and taking a network hop to an inference service, which adds a tail you do not control.
  • So the rule is: no feature computation and no RPC on the order path. A model co-located in the decisioner reading precomputed features is fine and fits the budget, which is a better answer than banning models outright.
• The score being minutes stale is accepted, out loud. Staleness is what the stream in the next section exists to cover, and pretending the inline path is authoritative is how designs end up with a 200 ms order.
• The decision log is written for the regulator, not for the engineer. Inputs, rule version, score version, verdict, latency. Immutable, 7 years, roughly 130 TB.

2) Near-real-time detection over the event stream

• Everything publishes to one stream: orders, logins, device changes, transfers, beneficiary additions. One stream rather than several is what makes cross-signal patterns expressible at all, and the account takeover sequence is precisely a cross-signal pattern.
• Carry order MODIFICATIONS and CANCELS, not just fills, and say why explicitly. Spoofing and layering are patterns in orders that were never meant to execute, so a feed of fills alone cannot see them. Adding cancel events roughly triples the stream volume and is the price of being able to answer a manipulation question at all.
• Windowed aggregates do the work the inline path had no time for: orders per minute per account, distinct instruments per hour, logins per device, failed step-ups per IP.
• The correlator is the part people leave out.
  • It joins accounts that share a device fingerprint, an IP, or a beneficiary bank account.
  • A per-account rule cannot see a ring by construction, no matter how good it is, because each individual account looks ordinary.
• The output is two levers, and they are deliberately different in strength.
  • Open a case for an analyst, ranked by exposure, when the pattern is suggestive.
  • Pull the freeze lever automatically when it is unambiguous. That is defensible only because a freeze is reversible: an analyst can undo it in minutes, and the customer gets an apology rather than a loss.
• The loop closes back into the feature store. A confirmed pattern updates the score that the 10 ms path reads, so detection at second N changes inline behaviour at second N plus a few. That loop is the product.

3) The egress gate, where the system is allowed to be slow

• Four checks, any one of which can hold, and none of which would be affordable on the order path:
  • The settlement clock. Funds from an unsettled deposit are not withdrawable. This alone kills the deposit-buy-withdraw cycle.
  • Destination cool-off. A beneficiary added in the last 72 hours cannot receive a withdrawal, because "change the bank account, then withdraw" is the account takeover playbook.
  • Step-up authentication on a device that has not been seen before.
  • The current risk score and any open case, the same score the order path reads, used here with a much lower tolerance.
• Hold, do not reject. A hold is reversible and an analyst can release it in minutes. A rejection is a support ticket, a complaint, and a customer who now believes the platform lost their money.
• Size the hold rate, because every hold ends at a human. At a 1 percent hold rate that is 2,000 holds a day and roughly 20 analysts a shift at five minutes each; at 0.1 percent it is two people. The rate is therefore a staffing decision, and the queue is ranked by exposure so the scarce hour goes to the largest number.
• Be explicit about instant buying power, because it is the interesting product tension. Letting a customer trade against an unsettled deposit is the platform lending its own balance sheet, capped, knowingly, and priced as a fraud loss. It is a business decision the system implements, not an oversight.
• The ACH return window outlasts everything. A deposit can be pulled back for up to 60 days, long after the trade settled and the customer left. That is why exposure is capped per account rather than merely delayed.

4) Offline: the graph, the labels, and the regulator

• Entity resolution links accounts by what they share: device, IP, bank account, beneficiary, address. The output is a graph, and fraud rings are communities in it.
• The UNLINKABLE ring is only visible here. A pair that shares a device or a bank account is caught by the correlator in seconds; what needs the graph is the ring whose only shared attribute is behaviour. Either way there is no per-order rule that catches it, because each account is individually ordinary.
• Self-trade prevention splits into two problems with two homes, and the mechanism is not mine even though the obligation is.
  • Same account both sides: tag the order with a self-match-prevention identifier so the VENUE cancels one side, plus an inline check against that account's own resting orders. An inline check alone only sees what routes through me, and the resting order may be at another venue.
  • FINRA Rule 5210.02 makes it my obligation to have policies reasonably designed to prevent self-trades across accounts under common beneficial ownership, which is why the graph below is a compliance artefact and not just a fraud tool.
  • Three accounts with one beneficial owner: a graph query that runs overnight.
• Training labels come from closed cases, not from guesses. An analyst's confirmed outcome is the only clean label the system produces, which makes the case queue a data pipeline as much as a workflow.
• Compliance files a SAR on a confirmed case, and the customer is not told. Hours of latency is acceptable here precisely because the egress gate already stopped the money moving.

1) What can you actually decide in 10 ms?

2) Fail open or fail closed?

3) Catching collusion between accounts

4) The deposit-buy-withdraw cycle

5) False positives, and the loop that fixes them

6) Manipulation: spoofing, layering, and marking the close

7) Account takeover, which is where the losses actually are

Four more dives, briefly

• The insider, which this design currently does nothing about and should.
  • Look at the API again: freeze, resolve a case, promote a rule. The dangerous direction is not freeze, it is UNFREEZE and RELEASE, and one analyst with the console can release every held withdrawal in the queue.
  • So: a second approver for any release above a notional threshold, rule promotion to enforcing approved by someone other than its author, and no standing production access.
  • Analyst actions land in the same WORM log as the decisions, and an analyst who releases an unusual share of holds is itself a monitored signal. The fraud system needs a fraud system.
• Adversarial drift, which is what makes this different from ordinary ML.
  • The adversary reads your decisions. Every block teaches them where the line is, and a stable model is a solved puzzle.
  • So: retrain often, and never expose the reason for a block precisely enough to probe. Keep any randomness in WHICH ambiguous cases get stepped up, never in whether a mandatory control fires, because non-deterministic enforcement is indefensible in an examination.
  • Watch for score distribution drift as an alarm in itself. A sudden change in the shape of the distribution usually means either a bug or a new attack, and you want to know which within the hour.
• Regulatory obligations, which shape the system more than people expect.
  • SAR filing has deadlines and the customer must not be told, so "tipping off" is a product constraint, not only a legal one.
  • Decisions are immutable and retained 7 years, which is why the decision log is WORM storage rather than a table someone can update. Since the 2022 amendments 17a-4(f) also allows an audit-trail alternative, so WORM is one compliant answer rather than the only one.
  • Be precise with "adverse action": refusing an order is not an ECOA action, but denying or cutting instant buying power is, because that is credit. It needs a stated reason, so a model that cannot explain itself cannot be its sole basis, which is why the rules layer survives alongside the model.
  • That squares with the anti-probing rule above: the customer gets a coarse reason, and the full reasoning sits behind the decisionId for support, the analyst and the regulator.
• The vectors I have not spent a dive on, named so the interviewer knows they were chosen against rather than missed.
  • Laundering through trading is the typology unique to this system: deposit dirty money, buy and immediately sell something liquid, and withdraw brokerage proceeds, paying the spread as the fee. It is why the egress gate looks at the RELATIONSHIP between deposits and trading, not just at the withdrawal. Structuring below reporting thresholds and sanctions and PEP screening sit here too.
  • Other funding rails have different clocks. Card deposits carry chargeback rights out to 120 days under network rules, longer than ACH. Crypto withdrawal is instant and absolutely irreversible, needs chain analytics and Travel Rule handling, and is the sharpest possible illustration of this page's own thesis.
  • Bank-account ownership verification is the cheapest deposit control there is: a name match through an aggregator, or micro-deposits. It stops funding from an account the customer does not own before any of the later machinery has to.
  • Promo and referral abuse is a top retail loss line and multi-accounting is the whole attack, so the entity-resolution graph already built is the answer with no new machinery.
  • Synthetic identities mature inside the platform over months, which is why "IDV is the vendor's problem" is only half true: the graph re-checks identity after onboarding, and "these accounts all opened in the same week and share an address" is where synthetics actually surface.

Final Design

• The order gateway binds a session to a device, asks the risk decisioner for a verdict, and gets allow, step-up or block within 10 ms from a precomputed score plus deterministic rules.
• Every order, login, device change and transfer publishes to one event stream, which is what makes cross-signal patterns expressible.
• Stream rules keep rolling counts the inline path has no time for, and a correlator joins accounts that share a device, an IP or a beneficiary.
• Detection produces two levers of deliberately different strength: a case for an analyst, and an automatic freeze that is safe only because it is reversible.
• The egress gate holds withdrawals on the settlement clock, the beneficiary cool-off, step-up, and the current score, and it fails closed.
• Offline, entity resolution and graph analytics find the rings, closed cases become training labels, and the retrained score is published back to the store the 10 ms path reads.
• The invariant worth closing on: the discretionary part of the order path fails open and the egress path fails closed, because an order's loss is bounded and a payout's is total. Every other decision on this page follows from that one.

What is Expected at Each Level

• Mid
  • Builds a risk service on the order path with rules and a score, plus a queue for analyst review.
  • Names the common fraud types and writes sensible rules for them.
  • Usually tries to do the detection inline and does not notice the latency budget is already spent.
  • Usually treats fraud prevention as blocking bad orders, and never gets to the money leaving.
• Senior
  • Separates the tiers by latency and puts the expensive work off the request path deliberately.
  • Identifies withdrawal as the irreversible step and gates it harder than trading.
  • Handles the deposit settlement clock and the beneficiary cool-off without being prompted.
  • Talks about false positives as a cost rather than a tuning parameter, and knows what shadow mode is for.
• Staff
  • Leads with the two-rate comparison and the reversibility asymmetry, and uses them as the reason for at least three separate decisions.
  • Answers "fail open or closed" by rejecting the question and splitting it per action, then gives the staged degradation.
  • Knows that cross-account fraud is a graph problem and that no amount of per-order cleverness substitutes, and can say which tier each variant belongs in.
  • Treats the analyst loop as a data pipeline: closed cases are the only clean labels, so queue ranking and analyst throughput are modelling constraints.
  • Raises the adversarial problem unprompted: the attacker reads your decisions, so a static model is a solved puzzle.
  • Knows the regulatory shape, including that you may not tell the customer, and that a model which cannot explain itself cannot be the sole basis for an adverse action.
  • Names what the design does not do: novel fast attacks get their first orders through, sophisticated rings that share nothing are found late if at all, and instant buying power is a priced loss.