← all topics UGC assets board · say it in this order 1 Problem2 Entities + API 3 High-level design4 Deep dives 5 Final + levelsreset

Build a content delivery system for avatars and skins

50M creator assets · 46K requests a second · 1.2 PB a day to clients · Hello Interview flow, board rules, one idea per line
SAY THIS SENTENCE FIRST

"The question offers me a trade-off, faster delivery against cache invalidation complexity, and I want to refuse it, because that trade only exists if I choose mutable URLs. If an asset is named by the hash of its own bytes then a new version is a new URL, every edge can cache for a year, and there is nothing to invalidate. I still take that trade exactly once, on the small pointer that says which version is current, because it is the only object cheap enough to afford it. What that buys me is the freedom to spend the whole hour on the two problems it does not solve: user generated content has a fifty-million-item cold tail rather than a hot working set, and moderation still needs one asset gone from two hundred edges in sixty seconds."

STEP 1 OF 5

Understanding the problem

5 min

Functional Requirements

  1. A client should fetch every asset it needs to draw an avatar, quickly, anywhere in the world.
  2. A creator should be able to publish a new version of an item, and players should get it.
  3. A moderator should be able to take an asset down, and it should stop being served everywhere within about a minute.
  4. It should work for the millions of items almost nobody requests, not only the popular ones.

Below the line (out of scope):

  • The authoring and upload pipeline, and the format conversion behind it. I care that ingest produces immutable named blobs; how a mesh gets optimised is a different hour.
  • The marketplace, pricing and the economy. Entitlement shows up once, in the API section, because it changes where authorisation lives, and then it goes away.
  • Rendering, and the moderation DECISION itself. Classifying an asset as banned is a machine-learning problem; propagating that ban is a delivery problem, and only the second one is mine.

Non-Functional Requirements

Do the scale math first, because it shows where the simple design breaks
  • The catalogue is tiny (15 TB) and the traffic is enormous (1.2 PB a day), so this is a caching problem, not a storage problem. Do the shape of the catalogue first, because it is the opposite of what a CDN expects.
    • 50M creator-made assets, averaging about 300 KB, is roughly 15 TB at origin. That is small. Storage is not the problem here.
    • 20M daily users fetching about 200 DISTINCT assets a session is 4 billion requests a day: about 46,000 a second, call it 139,000 at peak with a 3x diurnal factor that is conservative for a globally spread player base.
    • Say distinct, and say why, because the raw number is much larger. A single crowded scene naively wants 1,500 fetches (100 avatars times 15 assets). Almost all of that collapses: players share default bodies and popular items, and the client already holds most of it from earlier in the session. The 200 figure is the deduplicated working set, and the 1,500 is what you would fetch if nothing were done about avatar composition.
    • At 300 KB that is 1.2 PB a day leaving the edge. So the catalogue is tiny and the traffic is enormous, which tells you immediately that this is a caching problem and not a storage problem.
  • About ten thousand popular items carry most of the traffic, and the tens of millions of others form a tail with no hot working set. This distribution is what makes user generated content (UGC) different from a normal CDN workload.
    • They will sit hot in every PoP.
    • The other tens of millions are requested rarely, unpredictably, and from anywhere. A normal CDN assumes a hot working set; user generated content does not have one.
    • That tail is a minority of requests and nearly all of the origin load, which is the number the design has to attack.
  1. Latency p99 under about 100 ms, which forces the cache hit rate above 99 percent. Notice immediately that latency constrains the hit rate rather than being independent of it.
    • At a 98 percent hit rate the 98th percentile fetch IS a miss, so p99 is an origin round trip and the target is unreachable. Holding p99 at the edge needs better than 99 percent, realistically 99.5 and up once misses at every cache tier are counted.
    • The number that matters to a player is worse than the per-asset one, because an avatar is 15 assets: at 98 percent per asset, 26 percent of avatars contain at least one miss. At 99.5 percent it is 7 percent, at 99.9 percent it is 1.5.
    • So hit rate is a latency argument before it is a cost argument.
  2. Cache hit rate targets 99.5 percent or better, and it moves origin fill, not CLIENT egress. The 1.2 PB a day going to clients is fixed by the product. What it changes is origin fill and fleet size (24 TB a day at 98 percent, 6 TB at 99.5, 1.2 TB at 99.9), and the avatar-level miss rate above.
  3. Takedown propagation under about 60 seconds, globally. This directly contradicts caching for a year, and that contradiction is the most interesting thing on the page rather than an oversight.
  4. Publish visibility of seconds to a minute is fine. Creators tolerate a short delay; players never notice one.
  5. Graceful degradation: a failed asset fetch should render a placeholder, not block the scene. Availability here is about never blocking a frame, not about never missing bytes.
DONE WHEN: the interviewer has heard "15 TB of catalogue and 1.2 PB a day of traffic" and "there is no hot working set". Those two license the whole design.
STEP 2 OF 5

Entities + API

5 min

Defining the Core Entities

  • Asset is a blob of bytes named by their own hash: contentHash, bytes, mime type. It is immutable by construction, because changing the bytes changes the name.
    • The server computes that hash, never the client, and this is a security property rather than a detail. If an uploader can declare the digest, they can store bytes under someone else's hash and poison a year-TTL entry in 200 PoPs, in a system whose whole design has no purge path. Ingest rehashes what it received and rejects a mismatch.
    • This is the decision the whole design rests on, so say it in the entity list rather than saving it for a dive.
  • Item is what a creator and a player think of as "a hat": itemId, plus a pointer to the current contentHash and the history of previous ones. The item is mutable; the asset never is.
  • Variant is one rendition of an item for one target: a level of detail, a texture format, a platform. Each variant is its own asset with its own hash, because they are genuinely different bytes.
  • Composite: a baked low-detail avatar, keyed by inputHash.
  • Manifest is the mutable pointer: avatar id to a list of slot and content hash. In a session the client is PUSHED this by the session server; the HTTP object below is the cold-entry and out-of-session path.
    • Size it rather than calling it small. Fifteen slots with full URLs and sha256 hashes is about 2 KB. Truncating the hash to 128 bits, still far beyond collision risk here, gets it to roughly 1.5 KB. Bare hashes with no URLs would reach about 600 bytes, but that hands URL construction back to the client, which the API section deliberately refuses. Either way it is cheap in bytes; none of them is 400.
    • It changes whenever the player changes clothes, which is rare per player and constant in aggregate.
  • DenyList is the set of banned content hashes.

API or System Interface

WS   roster.push                     // session server -> client, the common path
                                     // { avatarId: [ { slot, url } ], ... } for the whole instance

GET  /v1/avatars/{userId}/manifest   // -> { slots: [ { slot, url } ] }, the fallback
                                     // Cache-Control: max-age=30

GET  /assets/{contentHash}.{ext}     // -> the bytes
                                     // Cache-Control: public, max-age=31536000, immutable

POST /v1/items/{itemId}/versions     // creator publishes -> { contentHash }
POST /v1/moderation/deny             // { contentHash } -> 202 Accepted
  • The asset is cached for a year and the manifest for 30 seconds, and that contrast IS the architecture. The two objects have deliberately opposite cache policies. One is about 2 KB and changes when a player changes clothes. The other is 300 KB, can never change, and is cached for a year. Giving both the same policy is the mistake the question is hinting at. The small one mostly does not travel over HTTP at all, which is the next step of the same argument.
  • No ETag on the asset, because the URL is the version. Conditional requests exist to ask "has this changed", and the answer here is structurally always no.
  • The manifest returns URLs, not ids. The client should never construct a URL, so the server can move assets, change variant selection or point at a different CDN without shipping a client.
  • Publishing returns a hash, not a success flag. The creator's new version exists the moment the bytes are stored; making it visible is a separate, tiny write to the item pointer.
  • Entitlement is enforced at the MANIFEST, never at the CDN, and be careful with the reason, because the obvious one is wrong.
    • Signed URLs do NOT cost you cacheability: CDN token auth deliberately excludes the signature from the cache key, so the object still caches once for everyone entitled.
    • The real reason is that the bytes are not the secret. Ownership is. A cosmetic asset leaks from the client the first time it renders, so signing adds key management and clock skew to protect something that is not protectable.
    • That would change if the asset were genuinely confidential, and saying so is what makes it a decision rather than a habit.
DONE WHEN: you have said "max-age 30 on one and a year on the other" and "the URL is the version, so there is no ETag".
STEP 3 OF 5

High-level design

10 min, end to end, no dives yet

1) Fetch every asset for an avatar

1) Fetch every asset for an avatar: a tiny mutable pointer, then immutable bytes In a session SessionServer pushes GameClient the roster of asset URLs; out of session GameClient asks ManifestService for an avatar's manifest, about 2 KB with a 30 second TTL that lists asset URLs by content hash. It then requests each asset from EdgeCache, 200 PoPs that keep the bytes for a year because a hash never changes. EdgeCache fills from the content-addressed AssetStore only the first time a PoP asks. 1) Fetch every asset for an avatar: a tiny mutable pointer, then immutable bytes the manifest changes and is small. the bytes never change and are enormous. give them opposite cache policies. the mutable half about 2 KB, changes on a wardrobe edit,30 s TTL ManifestService which asset versions doesthis avatar usesmall JSON, changes often the numbers this shape has to survive: 20M daily users, 200 distinct assets a session46,000 asset requests a second on average139,000 a second at peak, 1.2 PB a day out50M assets but only about 15 TB at originso the edge is the system, origin is a detail GameClient needs 15 assetsto draw one avatar20M daily users the immutable half about 300 KB each, never changes, one year TTL, never purged on publish EdgeCache 200 PoPs, the CDN tier139K asset requests a secondat peak, 1.2 PB a day outcached a year, never purged ON PUBLISH AssetStore content-addressed blobs50M assets, about 15 TBwritten once, never overwrittenthe hash IS the name 1. out of session, GameClient asks for the avatar's manifest 2. ManifestService returns a list of asset URLs with a 30 second TTL 3. GameClient requests each asset by its content hash 4. EdgeCache serves the bytes, kept a year because a hash never changes 5. EdgeCache fills from AssetStore only the first time THIS PoP asks SessionServer socket to every clientin the instance 1b. in a session, SessionServer pushes the roster of asset URLs instead, delta on change the manifest response GET /v1/avatars/8812/manifestCache-Control: max-age=30{ "head": "/assets/sha256:a91f....glb", "torso": "/assets/sha256:4c02....glb", "hat": "/assets/sha256:7e13....glb" } // 15 slots of URLs, about 2 KB the asset response GET /assets/sha256:a91f....glbCache-Control: public, max-age=31536000, immutableETag not needed: the URL IS the version. one object is about 2 KB and changes whena player redresses.the other is 300 KB and can never change.the same cache policy for both is the mistake. the not-so-happy paths: a manifest fetch that fails renders the player as a placeholder, which is bad but not fatal. an asset fetch that fails does the same.the manifest is the only thing a publish has to invalidate, and it expires on its own in 30 seconds.
  • There are two kinds of object: one small mutable pointer saying which versions are current, and many large immutable assets named by the hash of their bytes. Separating them is most of the design.
  • In a session, the session server pushes the roster: every avatar in the instance with its asset URLs, at spawn, and a delta when someone changes clothes. It already knows who is in the instance and already holds a socket to every client.
  • Out of session, the client fetches the manifest over HTTP: about 2 KB with a 30 second TTL. That covers cold entry, a profile page, and the push path being unavailable. It expires on its own, so nothing has to be purged.
  • The client then fetches each asset by its URL from the nearest PoP (CDN edge location), which fills from the origin object store on a miss. Assets are cached for a year, because a content hash can never point at different bytes.
  • Failure is a placeholder, not an error dialogue. A missing asset renders as a default shape; the scene keeps running. That is what lets the whole path be aggressive about caching rather than defensive about correctness.

2) Publish a new version

Publish a new version: store the bytes under their hash, then move the item pointer Creator uploads new bytes to IngestService, which hashes them on the server and stores the blob in AssetStore under that hash, then moves the item's pointer in ItemStore to the new hash. ManifestService reads the current hash when a client's 30 second TTL runs out; in a session SessionServer reads the current hash when it builds the next roster and pushes it instead. GameClient then requests the new URL from EdgeCache, which fills from AssetStore on its first miss. The old URL is not touched. 2) Publish a new version: store the bytes under their hash, then move the item pointer POST /v1/items/{itemId}/versions -> { contentHash }. storing the bytes and making them visible are two separate writes. Creator uploads a newversion of a hat IngestService hashes the byteson the server AssetStore origin object storeblob per content hashnever overwritten ItemStore itemId -> current hash+ history ManifestService 30 s TTL per avatar EdgeCache 200 PoPscached a year SessionServer socket to every clientin the instance GameClient draws the avatar 1. Creator uploads bytes 2. IngestService stores the blob under the hash it computed sha256:7e13... 3. IngestService moves the item pointer 4. ManifestService reads the current hash 5. ManifestService returns the new URL when the client's 30 s TTL runs out 4b. SessionServer reads the current hash for its next roster 5b. in a session, SessionServer pushes the next roster instead 6. GameClient requests the new URL 7. EdgeCache fills from AssetStore on its first miss, keeps it a year the item pointer, before and after: hat_42: sha256:a91f... -> sha256:7e13...history: [ sha256:a91f... ]/assets/sha256:a91f....glb still cached, nobody asks for it the not-so-happy paths: - the new version is a new URL, and the old one is left alone.- every version is stored forever, so storage only grows; the garbage collection dive handles it.
  • The creator uploads the new bytes; ingest hashes them on the server and stores the blob in the origin object store under that hash. POST /v1/items/{itemId}/versions returns the hash.
  • Making the version visible is a separate, small write: the item's pointer moves to the new hash, and the old hash stays in its history.
  • Players get the new URL from the next roster, which the session server builds from the item's current hash, or when their manifest's 30 second TTL runs out. The new version is a new URL. The old URL is still valid and still cached; nobody asks for it any more.
  • Every version is stored forever, so storage only grows; the garbage collection dive handles it.

3) Serve the items almost nobody requests

Serve the items almost nobody requests: the head stays hot, the tail misses to origin A popularity curve: about 10,000 head items take most of the requests and stay hot in every PoP, while the other 50M items share a minority of requests and almost always miss. A GameClient asks its PoP for a tail asset, the PoP misses and fetches it from Origin, and the client draws a placeholder meanwhile. Because a cold asset is cold in all 200 PoPs at once, one item can cost 200 origin fetches. A table shows origin requests and fill at four hit rates. 3) Serve the items almost nobody requests: the head stays hot, the tail misses a normal CDN assumes a hot working set. user generated content does not have one. share of requests top 10K top 1M 50M assets, ranked by popularity the head: about 10,000 itemsmost of all requestscached in every PoPhit rate near 100 percent the tail: the other 50M itemsalmost every request is a cold edge miss almost 50M assets live alongthis flat line the tail is a minority of requestsand it generates nearly all of the origin load. the simple design: every PoP fills from the origin on a miss GameClient needs a tail asset PoP 7 one of 200 edgelocations, a miss Origin object store,every asset 1. GameClient asks its PoP for a tail asset 2. PoP 7 misses and fetches the bytes from Origin 3. Origin returns the bytes, and PoP 7 keeps them 4. PoP 7 serves it; meanwhile the client draws a placeholder the weak spot: a cold asset is cold in all 200 PoPs at once Origin: 200 separate fetches for the same bytes PoP 1 PoP 2 PoP 3 ... PoP 199 PoP 200 each PoP that gets a request misses on its own:one item can cause 200 origin fetches for the same bytes,and the tail is where the origin load comes from.the cold-tail dive handles it. the same 46,000 req/s and 1.2 PB a day, at three cache hit rates cache hit rate origin requests a second origin fill a day 98 percent 926 req/s 24 TB 99.5 percent, the target 231 req/s 6 TB 99.9 percent 46 req/s 1.2 TB red misses the latency target: at 98 percent the 98th percentile fetch is a miss. client egress is 1.2 PB a day whatever you do;hit rate does not change what you ship to players. what it changes is origin fill and the sizeof the origin fleet. the not-so-happy paths: - a tail asset is a miss at almost every PoP that sees it, so its first load costs an origin round trip.- the client never blocks on it: a placeholder renders until the bytes arrive.
  • The head takes care of itself: the popular items stay hot in every PoP. The ten thousand or so popular items serve at nearly 100 percent hit rate without anyone doing anything clever.
  • A tail asset still loads: the PoP misses, fills from the origin, and the client draws a placeholder until the bytes arrive.
  • This is where the simple design strains: a cold asset is cold in all 200 PoPs at once, so one item can cause 200 origin fetches for the same bytes; the cold-tail dive handles it.

4) Take an asset down

Take an asset down: drop it from manifests and purge it from the CDN A Moderator bans a content hash through TakedownService, which records the hash in the DenyList, drops it from the item and every manifest so no new client learns the URL, and asks the CDN to purge the URL from all 200 EdgeCache PoPs. A Player holding the direct URL still gets the bytes from any PoP the purge has not reached yet. 4) Take an asset down: drop it from manifests and purge it from the CDN POST /v1/moderation/deny { contentHash }. the design says 'never invalidate'; moderation needs 'gone everywhere in 60 seconds'. the problem GET /assets/sha256:a91f2c...9d4b.png cached in 200 PoPs, never revalidatedCache-Control: public, max-age=31536000, immutable the design says this is immutable and cacheable forever.it is also, as of one minute ago, banned. it has to stop being served from all 200 PoPswithin about a minute, including to anyonealready holding the direct URL. the simple design Moderator TakedownService one moderator action DenyList the set of bannedcontent hashes ManifestService hands out asset URLs EdgeCache 200 PoPscached a year Player holding thedirect URL 1. Moderator bans a hash the 60 second clockstarts here 2. TakedownService records the hash in the DenyList 3. TakedownService drops the hash from every manifest 4. TakedownService has the CDN purge the URL 5. Player requests the direct URL; a PoP the purge has not reached still returns 200 a purge across 200 PoPs is slow and best-effort, and the URL keeps servinguntil it lands everywhere. the takedown dive handles it.
  • A moderator bans a content hash through the takedown service (POST /v1/moderation/deny), which records the hash in the DenyList.
  • The takedown service drops the hash from the item and from every manifest, so no new client learns the URL.
  • It asks the CDN to purge the URL from all 200 PoPs, so they drop the cached bytes.
  • This is where the immutable design bites back: an asset cached for a year must be gone from 200 edges in under a minute, and a purge across 200 PoPs is slow and best-effort while anyone holding the direct URL keeps getting the bytes; the takedown dive handles it.
DONE WHEN: all four requirements have a path, and you have said out loud that takedown is the one place the immutable design makes life harder.
STEP 4 OF 5

Potential deep dives

~20 min, interviewer steers
TRIGGER: "how do you handle cache invalidation" / "what about the trade-off"

1) The trade-off in the question, and why I will not take it

IMMUTABLE URLS · MUTABILITY CONFINED TO A POINTER · NOTHING TO PURGE
Bad Solution: mutable URLs and a purge API
  • /assets/hat_42.glb keeps its name and gets new bytes, and you call purge on publish.
    • Now you need a purge fan-out to 200 PoPs, a retry story for the ones that fail, and an answer to "is it done yet" that nobody can give you confidently.
    • Because purge is unreliable you also shorten the TTL as a safety net, which lowers hit rate and raises origin load. That is precisely the trade-off the question describes, and you built it yourself.
    • You have also made the system non-deterministic: two players in different regions can see different hats for an unbounded window.
Good Solution: a version in the query string
  • /assets/hat_42.glb?v=7, bump the version on publish. This genuinely works and is what most sites do.
    • The weakness is that the version is metadata someone has to maintain, and it is easy to forget to bump, which produces a stale asset with no error anywhere.
    • Some caches historically treated query strings inconsistently, which is a smaller problem now but still a reason to prefer the path.
    • It also does not deduplicate: the same bytes uploaded by two creators are two cache entries and two origin objects.
Great Solution: name the bytes by their own hash
  • The URL contains the content hash, so the name and the bytes cannot disagree. You cannot forget to bump a version, because the version is derived rather than declared.
  • Cache for a year, everywhere, with no purge path at all. The immutable directive also tells the browser not to revalidate on refresh, which removes a whole class of pointless conditional requests.
  • Deduplication is free. Two creators uploading identical bytes produce one object and one cache entry, which on a 50M item UGC catalogue with heavy template reuse is a real saving rather than a curiosity.
  • Rollback is free too, because the previous version was never deleted, so reverting is a pointer write. It is the part people miss.
  • The mutability moves to the manifest, which is cheap in bytes but not in requests. Cheap has to be measured in requests, not bytes. The manifest is now the thing that changes fast. At about 2 KB it is trivially cheaper than a 300 KB blob in 200 places, but a per-avatar object with a 30 second TTL barely caches, so in a crowded scene it would outnumber the asset traffic. That is why the session server pushes the roster and the CDN object is the fallback.
    • Your own manifest, refreshed on a 30 second TTL, is about 14,000 requests a second: 20M daily users each playing one 30 minute session averages about 420,000 players online at once, and 420,000 refreshes every 30 seconds is 14,000 a second. Affordable, but already a third of the asset rate.
    • Other players' manifests are the problem. A scene shows other avatars, and each one is a manifest. At a million concurrent players with 100 visible avatars on a 30 second TTL that is over 3 million requests a second, roughly seventy times the asset traffic the entire design is sized around.
    • And it caches badly, which is the part that turns it from expensive into dangerous. A per-avatar key has about one reader, so a 30 second TTL buys almost no edge hits. Manifest misses, not asset misses, would become the dominant origin load.
    • So the session server, which already holds a socket to every client, pushes the hash list at spawn and a delta on change. Avatar equipment is game state, and shipping game state over a CDN was the mistake.
    • The CDN manifest object stays, for cold entry, for out-of-session views like a profile page, and as the fallback when the push path is unavailable. It is then a small fraction of that rate rather than all of it.
  • Name the two costs honestly: storage grows forever and needs a collector, and a URL that leaks is valid forever, which is what makes the takedown dive necessary.
  • And concede where a mutable URL is genuinely right, because that is the obvious counter-question.
    • Anything a third party hotlinks: a profile picture in an embed, a shop listing image. There you want the address stable and the bytes to change, so content addressing forces a mutable redirector anyway.
    • Clients you cannot update, where an old binary has a path compiled in and can never learn a new hash.
    • The manifest itself, which IS a mutable URL bounded by a short TTL. So the accurate claim is not that the trade-off disappears, it is that I take it exactly once, on the one object small enough to afford it.
TRIGGER: "50 million assets" / "what about the unpopular ones"

2) The cold tail, which is what UGC actually means

A COLD ASSET IS COLD EVERYWHERE · SHIELD THE ORIGIN · COALESCE THE HERD
Bad Solution: rely on the edge cache
  • Put a CDN in front of the origin and assume the hit rate will be fine.
    • It is fine for the head and useless for the tail, and the tail is where the origin load comes from.
    • Worse, a rarely requested asset gets evicted between requests, so the same item can miss repeatedly at the same PoP and look like a permanently cold object.
    • Then every one of 200 PoPs independently discovers the same cold asset and independently asks origin for it.
Good Solution: raise the TTL and the cache size
  • Cache for longer and buy more edge storage, so fewer things fall out. Correct as far as it goes, and content addressing already gives you the long TTL for free.
    • But you cannot hold 50M assets in every PoP, and most of them would never be read if you did.
    • It does nothing at all about the fan-in: the first request for a cold asset still multiplies by the number of PoPs.
Great Solution: a tier between the edge and the origin, plus coalescing
  • Put a tier of regional shield caches between the edges and the origin, so a PoP that misses asks its shield, not the origin. Then the 200-way fan-in becomes one fetch per shield: roughly 10 with a handful of regions, not 1. And since the whole catalogue is 15 TB for one rendition and about 180 TB for all twelve, it fits in a shield either way, so the shields converge on holding everything and the tail stops being a tail.
    • A single global shield is the wrong shortcut: it is a single point of failure for all origin fill and adds a transcontinental hop to a 100 ms budget.
    • The fan-in is also worse than 200, which is the detail that shows you have run one. A PoP is many cache nodes, so without consistent hashing inside the PoP you multiply by nodes per PoP. That intra-PoP layer is a real component, not a setting.
  • Coalesce concurrent misses inside each PoP, so ten thousand simultaneous requests for one cold asset produce a single upstream fill. The rest wait on it, which is a config setting rather than a design, and forgetting it turns a popular drop into a self-inflicted denial of service.
  • Edges keep a small hot set with ordinary eviction, and the shields hold essentially everything. The two tiers have different policies. That is what makes the tail affordable without putting the whole catalogue in all 200 PoPs.
  • At 46,000 requests a second, a 98 percent hit rate sends 926 a second to origin and 99.9 percent sends 46. That is 24 TB a day of fill against 1.2 TB. The shield is what moves you between those rows.
    • Hit rate is a latency argument before it is a cost argument, and that is the strongest case for the shield in the whole design.
  • Hit rate buys origin fleet size and fill cost, not client egress, which is 1.2 PB a day no matter what you do.
  • Prefetch the predictable tail. You often know what a client will need before it asks, because the roster names it, so the client can request the whole set at once and the edge can warm on entry rather than on the first miss.
  • Serve something before the whole thing arrives. Range requests over a mip-ordered texture let the client show a low-resolution version immediately and refine it, which serves the "never show a grey placeholder" goal better than the placeholder does, and pairs naturally with the LOD variants.
  • Close the loop on what to warm. Edge and shield miss telemetry is what turns "this hash is suddenly hot in one region" into a push to the others, which is the only way to pre-warm the drops nobody announced.
TRIGGER: "a popular creator drops an item" / "what happens at launch"

3) The hot drop, which is the opposite failure

COLD EVERYWHERE AT ONCE · PRE-WARM THE HEAD · COALESCE OR MELT
Bad Solution: publish and hope
  • Write the new version, let traffic warm the caches naturally.
    • A famous creator's drop is a million clients requesting one brand new hash within a minute, and that hash is cold in all 200 PoPs simultaneously.
    • Without coalescing, each PoP forwards every one of those requests, so origin sees a spike measured in hundreds of thousands of requests a second for a single object.
    • This is the one moment where the design's usual comfort, that the tail is rare, is exactly wrong.
Good Solution: rely on request coalescing
  • Let each PoP collapse concurrent misses into one upstream fetch. This is the essential mechanism and it fixes most of the spike.
    • You still get one fill per PoP, so origin sees 200 near-simultaneous requests rather than a million. Survivable, but it is a thundering herd you chose to accept rather than avoid.
    • And the first requester at every PoP eats the full origin round trip, which for a launch moment is exactly the user you least want to make wait.
Great Solution: warm it before anyone asks
  • Push the asset to the regional shield caches, and for a known launch to the edges, before the item is referenced by any manifest. Publishing the bytes and publishing the pointer are already separate steps, which makes this natural rather than a special case.
  • The bytes can be fully distributed while still invisible, because nothing points at them until the item pointer flips. That separation is the real payoff of content addressing here. Flipping the item pointer is then instant and cold-start free.
  • Keep request coalescing on as the backstop, because most drops are not scheduled and you will not pre-warm the ones you did not know about.
  • Rate limit the origin explicitly rather than hoping. The origin should shed load and let the shields serve stale-if-error rather than fall over, since a slightly old asset is a much better outcome than no asset.
  • The tail is many cold assets each asked for rarely; the drop is one cold asset asked for by everyone. Both are cache misses and they need different mitigations, which is why they are separate dives.
TRIGGER: "an asset gets banned" / "how do you remove something"

4) Takedown against a design that never invalidates

REFUSE, DO NOT PURGE · TWO MECHANISMS · A LEAKED URL IS FOREVER
Bad Solution: remove it from the manifest
  • Stop referencing the hash and consider it handled.
    • The bytes are still cached in 200 PoPs with a year of TTL, and the URL still works for anyone who has it.
    • For UGC takedowns, the people most likely to hold the direct URL are exactly the people you are removing it from.
    • It is also invisible: nothing in the system reports that the asset is still being served, so you will believe it worked.
Good Solution: purge it from the CDN
  • Call the purge API for that URL across every PoP. This is the obvious answer and it does eventually work.
    • Purge is slow and best-effort at 200-PoP scale, which is uncomfortable when the requirement is measured in seconds and the content is illegal.
    • It also has no memory: a request arriving after the purge but before the origin is updated can refill the very object you just removed.
    • And you have now reintroduced the purge infrastructure that content addressing let you delete, for one rare case.
Great Solution: a deny list checked at the edge, with purge as cleanup
  • Publish the banned hash to a small deny list that every PoP holds in memory, and check it before serving. Propagation is seconds because the object is tiny, and it is a push rather than a fan-out of per-URL operations.
    • Name the propagation mechanism rather than saying "a config push". It is an edge key-value store or a config activation, and the products differ enough to matter: some cap at a few megabytes, and some document propagation up to 60 seconds, which spends the whole takedown budget before the decision has even landed.
    • Decide the failure mode, because it is the most consequential policy here. A PoP without the current list either serves and alarms, or refuses and causes an outage. Pick one deliberately and write it down.
  • The refusal is the fast path; eviction is cleanup. The bytes can leave the cache lazily, because a served 410 Gone is what the requirement actually asks for, and 410 is the honest status for something deliberately and permanently removed.
  • Keep both mechanisms: the manifest stops new references, and the deny list stops the leaked URL. Neither alone is sufficient and the interviewer is usually probing for exactly that.
  • Make the check cheap and correct: the exact deny list held in memory at every PoP. A bloom filter only helps once the list outgrows memory, and even then every positive needs a confirming lookup, because a false positive means refusing a legitimate asset to everyone.
    • Ban a fraction of a percent of a 50M catalogue and you have hundreds of thousands of hashes. At 32 bytes each that is single-digit megabytes, which fits in memory at every PoP, so use the exact set.
    • A bloom filter is ornamental here: where the exact set fits it buys nothing, and where it would help the confirming lookup becomes a network call on the request path.
  • The origin must refuse the hash too, so a refill cannot bring it back. Refill has to respect the deny list. Origin must also refuse the hash, or a cache miss after a lazy eviction quietly resurrects the content.
  • Design the UN-ban, because classifiers are wrong and appeals are routine. A 410 is heuristically cacheable, so reversing a ban would mean evicting those 410s, which is a purge, which is the machinery this design deleted. So mark the refusal no-store, make the deny list a set you can remove from rather than append-only, and reversal becomes another small push.
  • Cover bulk takedown, which is the common case and which content addressing makes harder. Banning a creator means banning thousands of assets, and hash-named URLs share no prefix, so there is no "remove everything by creator 8812" operation. Keep a creator-to-hash index for exactly that, and push the whole set as one update.
  • Own the residual honestly: a client that already downloaded the bytes still has them.
    • The operational version matters more: a running client keeps DRAWING a banned skin on other players until something tells it to stop, and the edge cannot reach a client that never requests. The roster push from the session server is what fixes that, and it beats any cache TTL.
TRIGGER: "one avatar is a lot of assets" / "a hundred players on screen"

5) Composition, and the request amplification nobody costs

ONE AVATAR IS 15 FETCHES · THE ROSTER IS THE BATCH · BAKE THE FAR ONES
Bad Solution: fetch each part as the renderer needs it
  • Ask for each mesh and texture when the renderer reaches it.
    • One avatar is roughly 15 assets, so 100 players in view is 1,500 requests before deduplication, arriving in a burst at the moment the scene loads.
    • Discovering them one at a time also serialises the round trips, so the slowest part of the scene is the number of hops rather than the bytes.
Good Solution: multiplex them
  • Use HTTP/2 or HTTP/3 so the 1,500 requests share a connection without head-of-line blocking. Genuinely helps, and it is the right transport.
    • It removes the connection cost but not the request count, and each one is still a separate cache lookup and a separate chance to miss.
    • The deeper issue is that the client does not know what it needs until it has parsed something, so it cannot start early.
Great Solution: let the manifest describe the whole set, then reduce the set
  • Deduplicate across players before requesting. A hundred players wearing the same default body is one asset, and the client should collapse the set rather than asking 100 times.
  • Bundle the common case into one hash, turning the most frequent 15 fetches into one. The default body and the starter items ship as one bundle with one hash, which turns the most frequent path from 15 fetches into one.
  • Bake distant avatars server-side into one low-detail composite. A player 80 metres away does not need 15 separate assets; they need one low-detail composite. Compositing that once, caching it under its own hash, and serving it to everyone is a much better trade than shipping parts nobody can see.
  • The ROSTER, pushed by the session server, lists every hash the scene needs, so the client learns the full working set in one message and can fetch with full parallelism immediately, rather than discovering dependencies as it renders. That is the session server's push, not a manifest fetch: a manifest is per avatar, so a hundred-avatar scene would otherwise be a hundred of them.
  • Key the cached composite on a hash of its inputs, and do not overclaim how often it is reused.
    • The key is a hash of the INPUTS, the sorted component hashes plus LOD and format, because the output hash is not known until after baking. That means a small input-to-output mapping table, which is another mutable pointer and belongs in the entity list rather than being waved at.
    • Do not overclaim the reuse. Combination entropy adds across slots, so even one effective bit per slot over 15 slots is 32,768 composites and two bits is a billion. The real win is narrower and still worth it: the default and near-default avatars, which are a large share of any crowd.
    • Baking is also a service with a fleet and a queue, not a property. If it is the answer for the far case it has to appear in the architecture.
  • Say the cost: baking adds a build step and a new class of derived asset to garbage collect, and it is only worth it because the far case is the common case in a crowded scene.

Three more dives, briefly

  • Variants turn one item into about twelve assets, which multiplies storage and the cold tail by twelve. It is the combinatorial explosion nobody plans for.
    • One logical item becomes several real assets: levels of detail, texture formats per GPU family, platform-specific meshes. Three LODs times four formats is twelve objects per item.
    • Each variant is its own hash, which keeps immutability intact, but it multiplies both storage and the cold-tail problem by the same factor.
    • So correct the headline number when you say this, or the interviewer will. 50M items at twelve renditions is 600M cacheable objects and roughly 180 TB, not the 15 TB quoted earlier, which was one rendition each. The conclusion survives easily, since 180 TB is still small, but the figure has to move with the claim.
    • So generate variants lazily for the tail and eagerly for the head, and let the manifest decide which variant a given client is told about, because that keeps the decision on the server where it can change.
  • Nothing is ever overwritten, so garbage collection by reference is mandatory.
    • Nothing is ever overwritten, so storage only grows: old versions, abandoned uploads, derived variants and baked composites all accumulate.
    • Collect by reference: a blob is deletable when no item version, no manifest and no retention policy points at it, and moderation history has to be treated as a reference or you will delete evidence.
    • The safety bound is the client's retention, not the manifest TTL, and getting that wrong deletes live blobs. A client receives its roster at scene load and holds that working set for the whole session, which is minutes to hours, not 30 seconds. Size the grace window against session length and then some.
    • The race that content addressing creates is on the WRITE side and it is easy to miss. A creator uploads bytes that dedup onto an existing blob, and the collector deletes that blob between the dedup hit and the pointer write, so the publish points at nothing. Take a lease or a reference at dedup time, or use a mark epoch that aborts if any new reference appeared while marking.
    • The client needs a collector too, for the same reason the server does. Hash-named assets never expire, so a game client accumulates them forever unless it has a disk budget and an eviction policy. Immutability makes client-side GC mandatory, not optional.
    • Deletion must be slow and reversible, because content addressing means a mistaken delete cannot be repaired by re-uploading something similar. Only the identical bytes restore the URL.
  • Client egress of 1.2 PB a day is the dominant cost, and only compression, level of detail and bundle size move it. For a delivery system, cost is a first-class design input rather than an afterthought.
    • 1.2 PB a day of client egress is the dominant line and it is fixed by the product, so the levers are compression, level of detail and bundle size, not caching.
    • Caching buys origin fleet and fill, which is the smaller number, and it is worth being honest that hit rate is a capacity story more than a bill story here.
    • Price the requests, not only the bytes. 4 billion requests a day at typical CDN request pricing is on the order of a million dollars a year, roughly half the egress bill, and it is driven entirely by the request amplification of avatar composition. Bundling and baking are the only things that move it, which is the cost argument for them rather than the latency one.
    • Cash in the indirection you already built. Because the manifest hands out URLs rather than ids, steering a fraction of traffic to a second CDN is a server-side change with no client release. On a design betting 1.2 PB a day on one vendor, that is the cheapest availability insurance going, and it is already paid for.
    • Residency and erasure collide with immutability, and this is the sharpest legal edge here. "Stored forever" and "cached for a year in 200 PoPs" sit against an erasure obligation on user-generated content, and an EU creator's asset is replicated globally by default. The deny list is most of the enforcement; what is missing is a residency policy on where blobs may be filled from, and an erasure path that actually deletes rather than refuses.
    • The genuinely large saving on bytes is making them smaller: Brotli for meshes and JSON, GPU formats like ASTC or BCn that are already compressed and gain nothing from another pass, and geometry compression. Applied to the head of the distribution, that beats any caching change, because the head is most of the traffic.
    • Size the fleet at PEAK, not at the average. 139,000 a second is the number the origin, the shields and the manifest path have to survive, and quoting an average and then sizing nothing against the peak is a common way to be wrong by 3x.
DONE WHEN: you have opened two or three dives properly, and you have used "immutable" as a REASON at least twice rather than as a label.
FLASHCARDS · THE FIVE HARDEST PROBESshow all (interview mode)
The question offers faster delivery versus cache invalidation complexity. Do you take that trade?
  • No, and refusing it is the answer. That trade-off only exists if you chose mutable URLs.
  • Name the asset by the hash of its own bytes, and new bytes are a new URL. There is nothing to invalidate, so the TTL can be a year and hit rate is not in tension with freshness.
  • A version in the query string is the halfway answer: it works, but the version is metadata someone can forget to bump, and it does not deduplicate identical bytes.
  • The mutability does not vanish, it gets confined: about 2 KB rather than 300 KB in 200 PoPs. But cheap in bytes is not cheap in requests, so the roster travels over the session socket in the common case and the CDN manifest is the cold-entry fallback.
  • Free extras worth naming: deduplication across creators, and rollback as a pointer write because the old version was never deleted.
  • The two costs: storage grows forever and needs a collector, and a leaked URL is valid forever, which is why takedown needs its own mechanism.
  • Concede where mutable is right, because that is the counter-question: anything a third party hotlinks, clients you cannot update, and the manifest itself. The honest claim is that I take the trade once, on the one object small enough to afford it.
50 million assets and most are almost never requested. What breaks?
  • A normal CDN assumes a hot working set and UGC does not have one. The head is about ten thousand items and takes care of itself; the tail is where the origin load comes from.
  • The specific failure is fan-in: a cold asset is cold in all 200 PoPs at once, so one item can produce 200 independent origin fetches for identical bytes.
  • Regional shields collapse it: edges miss to a mid-tier that fills once for everyone behind it. With a handful of regions that is one fetch per shield, roughly 10 rather than 1, still a 20x cut and the honest number.
  • The win worth claiming: the whole catalogue is 15 TB, so it fits entirely inside every shield. That makes the shield close to a full replica rather than a hot-set cache, which is what makes 99.9 percent reachable.
  • The fan-in is worse than 200 unless you hash consistently INSIDE each PoP, because a PoP is many cache nodes.
  • Coalesce concurrent misses inside each PoP too, so 10,000 simultaneous requests for one cold asset produce one fill rather than 10,000.
  • Give the tiers different policies: small hot set at the edge, large warm set at the shield, so you never need 15 TB in every PoP.
  • The arithmetic: 46K req/s at 98 percent hit is 926 origin req/s and 24 TB a day of fill; at 99.9 percent it is 46 req/s and 1.2 TB.
A moderator bans an asset. Walk me through the next 60 seconds, given you cache for a year.
  • This is the one place the immutable design makes life harder, and it should be walked into deliberately rather than discovered.
  • Removing it from the manifest is necessary and not sufficient: it stops new references, but the direct URL still returns 200 from 200 PoPs for the rest of the year.
  • The fast path is a refusal, not a purge: publish the hash to a small deny list that every PoP holds in memory and checks before serving. It is tiny, so it propagates in seconds.
  • Evicting the bytes is cleanup and can be lazy, because a served 410 Gone is what the requirement actually asks for.
  • Origin must refuse the hash too, or a miss after a lazy eviction quietly refills it.
  • Make the check cheap and correct: the exact set in memory, because a false positive refuses a legitimate asset to everybody. A bloom filter with a confirming lookup only helps once the list outgrows memory.
  • Design the un-ban too: a 410 is heuristically cacheable, so mark the refusal no-store and make the deny list removable, or reversing a ban needs the purge machinery you just deleted.
  • Bulk is the common case: banning a creator means thousands of hashes with no shared prefix, so keep a creator-to-hash index and push the set in one update.
  • The residual you cannot fix: someone who already downloaded the bytes still has them. A running client also keeps DRAWING it on other players until the session server pushes a new roster, which beats any cache TTL.
One avatar is 15 assets and there are 100 players in view. How many requests?
  • Naively 1,500, arriving as a burst exactly when the scene loads, and serialised if the client discovers each dependency as it renders. That is the pre-deduplication number: the 200 distinct assets a session used for sizing already assume this work is done.
  • HTTP/2 or 3 removes the connection cost but not the request count, and each one is still a cache lookup and a chance to miss.
  • The roster push is the fix: the session server sends the whole working set in one message, so the client fetches with full parallelism immediately instead of discovering as it goes. Not a manifest fetch, because a manifest is per avatar and a hundred-avatar scene would be a hundred of them.
  • Then reduce the set: deduplicate across players, since a hundred players in the default body is one asset, and bundle the common starter case into a single hash.
  • Bake distant players server-side into one low-detail composite. That composite is content-addressed too, so a popular combination of items is one cached object shared by everyone wearing it.
  • Key the composite on a hash of its INPUTS, since the output hash is unknown until after baking. That input-to-output table is another mutable pointer and should be admitted as one.
  • Do not overclaim the reuse: combination entropy adds across slots, so even one bit per slot over 15 slots is 32,768 composites. The real win is the default and near-default avatars, which are a large share of any crowd.
What is your cache hit rate, and what is each point actually worth?
  • Target 99.5 percent or better, and be precise about what it buys, because the obvious answer is wrong.
  • The 1.2 PB a day going to clients is fixed by the product. Hit rate does not change what you ship to players at all.
  • What it moves is origin fill and origin fleet size: 24 TB a day at 98 percent, 6 TB at the 99.5 target, 1.2 TB at 99.9, and 926 against 231 against 46 origin requests a second.
  • But lead with latency, not capacity, because that is the bigger effect. An avatar is 15 assets, so at 98 percent per-asset hit, 26 percent of avatar loads contain at least one origin round trip. At 99.5 percent it is 7 percent, at 99.9 it is 1.5. That is what the player feels.
  • On cost, be careful in both directions: it is a capacity story rather than a client-egress story, but origin fill is not automatically free either. It is genuinely free on an S3-to-CloudFront path and very much not on most others, so name the origin before claiming the saving is a rounding error.
  • Content addressing is what makes a high hit rate reachable, because the TTL is a year and nothing forces it down for freshness.
  • Price the requests as well as the bytes: 4 billion a day is on the order of a million dollars a year, and bundling and baking are the only things that move it.
  • The genuinely large saving on bytes is making them smaller: better compression or a lower LOD applied to the head of the distribution beats any caching change, because the head is most of the traffic.
STEP 5 OF 5

Final design + what is expected at each level

wrap

Final Design

UGC asset delivery: immutable bytes, a mutable pointer, and one deny list A creator's upload is named by its hash, built into platform variants and written to AssetStore; publishing is IngestService pointing ItemVersions at the new hash. A player's client gets asset URLs from ManifestService, then fetches each asset by hash from the nearest EdgeCache PoP, which misses to a regional Shield that fills from AssetStore once. A moderator's ban lands in a DenyList that every PoP and the origin check, and VersionGC deletes blobs nothing references. UGC asset delivery: immutable bytes, a mutable pointer, and one deny list 20M daily users, 46,000 asset requests a second, 1.2 PB a day to clients, 50M assets that are only 15 TB at origin. PUBLISH how a creator's work gets in Creator IngestService hashes the bytes, that hash IS the name VariantBuilder LODs, texture formats,platform renditionseach variant is its own hash ItemVersions itemId to current hash, plus history 6. Creator uploads new work 7. IngestService names the bytes by their hash, so a new version cannot collide with the old then hands the named bytesto VariantBuilder 8. VariantBuilder produces the renditions each platform will actually ask for 9. IngestService points ItemVersions at the new hash, which is the whole publish step READ PATH what a player actually does GameClient 15 assets to drawone avatar ManifestService avatarId to asset URLs~2 KB, max-age 30, mostly PUSHED EdgeCache 200 PoPsmax-age one year, immutable Shield regional mid-tier200 cold misses become 1 per shield AssetStore content-addressed blobs50M assets, about 15 TB the two cache policies, side by side manifest: Cache-Control: max-age=30asset: Cache-Control: public, max-age=31536000, immutable the small mutable object expires on its own.the large immutable object never has to. 1. GameClient asks ManifestService which asset versions this avatar uses 2. ManifestService answers with asset URLs and a 30 second TTL (the fallback path) reads the current hash per itemfrom ItemVersions 3. GameClient requests each asset by hash and gets it from the nearest PoP 4. EdgeCache asks the Shield on a miss, once for all concurrent misses 5. Shield fills from AssetStore once, on behalf of every PoP behind it writes every renditioninto the AssetStore one avatar = 15 assets. a 100-avatar scene wants 100 rosters,which is why the session server pushes them, not the CDN. SAFETY the one true invalidation DenyList banned hashes, tiny, removableevery PoP and origin check it TakedownService Moderator LeakedUrlStillWorks immutable means a direct URL lives foreveronly an active refusal stops it 11. EdgeCache checks each hash against the DenyList and refuses a banned one, even though it still holds the bytes 10. Moderator bans a hash and TakedownService writes it to the DenyList a SET, not a log:appeals need un-ban too an active refusal isthe only thing thatstops a leaked URL ban 0.5 percent of 50M and the list is 250,000 hashes,about 8 MB: it still fits in memory at every PoP. what the refusal looks like at the edge: GET /assets/9f2a4c1b... -> 410 Gonethe bytes are still in the PoP, nothing was purged. COST AND SCALE what the numbers force EgressBudget 1.2 PB a day to clientsfixed by the product, not by hit rate OriginFill what hit rate actually moves24 TB a day at 98 percent VersionGC reclaims versions nothing referencesimmutability is not free the 2 percent that missesbecomes this 12. VersionGC deletes blobs that no item version, manifest or retention rule references any more the arithmetic: 46,000 requests/sec x about 300 KB = 13.9 GB/sec = 1.2 PB a day.2 percent of that misses the edge = 24 TB a day of origin fill.99 percent instead of 98 halves the fill to 12 TB. Egress does not move.50M assets x about 300 KB = 15 TB: the library is smaller than one day of egress. GC waits out the SESSION, not the manifest TTL:a client holds its working set for minutesto hours, not 30 s. The question frames this as faster delivery versus cache invalidation complexity. Content addressing refuses the trade: immutable URLs neverneed purging, and all the mutability is confined to a ~2 KB pointer that expires by itself.What it does not refuse: a leaked URL is forever, storage grows monotonically, and the long tail means a cold asset is cold in all 200 PoPs at once.
  • In a session the client gets its roster of content hashes pushed by the session server; out of session it fetches a manifest of about 2 KB with a 30 second TTL. Either way it then fetches every asset it names by content hash, cached for a year at the nearest of 200 PoPs.
  • Assets are named by the hash of their bytes, so publishing a new version creates a new URL and there is no purge path anywhere in the system.
  • Regional shields sit between the edges and the origin, so a cold asset costs one origin fill per shield, roughly 10 rather than 200, and each PoP coalesces concurrent misses into a single upstream request.
  • Ingest hashes the bytes, builds the variants each platform will ask for, and publishing is a pointer write from the item to the new hash.
  • Takedown publishes a hash to a small deny list that every PoP and the origin check before serving, because the manifest alone cannot stop a URL that already leaked.
  • A collector reclaims blobs that no item version, manifest or retention rule references, because nothing is ever overwritten.
  • The invariant worth closing on: the large objects are immutable and the small object is not, which is what turns the question's trade-off into a non-question. The two things it does not solve are the cold tail and the takedown, and those are the dives worth having.

What is Expected at Each Level

  • Mid:
    • Puts a CDN in front of an object store and serves assets from the edge.
    • Knows that versioning the URL avoids stale content.
    • Usually accepts the stated trade-off and designs a purge path for it.
    • Usually assumes a hot working set and does not notice the tail.
  • Senior:
    • Versions the asset URL, for example a ?v=7 bumped on publish, so players never get stale bytes, and knows the version is metadata someone can forget to bump.
    • Identifies the cold tail, raises the TTL and the edge cache size, and knows that does nothing about the fan-in: the first request for a cold asset still multiplies by the number of PoPs.
    • Purges the URL from every PoP as well as dropping it from the manifest, and knows purge is slow, best-effort and can be refilled.
    • On a hot drop, relies on request coalescing at each PoP, and knows it still leaves one fill per PoP, about 200 near-simultaneous origin requests.
    • On composition, sends an avatar's 15 parts over HTTP/2 or HTTP/3, and knows that removes the connection cost but not the request count.
    • Sizes the traffic and the catalogue instead of describing them qualitatively.
  • Staff:
    • Refuses the trade-off in the question explicitly, and can explain that it is an artefact of mutable URLs rather than a law.
    • Says where the mutability went instead of claiming it disappeared, and gives the small object its own policy and path.
    • Separates the two cache-miss problems: many cold assets each asked for rarely, and one cold asset asked for by everyone at a drop.
    • Is precise that hit rate moves origin fill and fleet size, not client egress, rather than overclaiming a cost saving.
    • Treats takedown as the one true invalidation and designs refusal rather than purge, including the origin refill path.
    • Raises composition and request amplification unprompted, and reaches server-side baking for the far case.
    • Names what immutability costs: monotonic storage, a mandatory collector, an irreversible delete, and a leaked URL that lives forever.
DONE WHEN: you can say, in one breath, hash-named bytes cached for a year, a shield for the tail, coalescing for the drop, and a deny list for the takedown, and then add the two things the diagram does not draw: the roster is pushed by the session server rather than polled over a CDN, and baking is a service with a fleet.