Build a content delivery system for avatars and skins

Functional Requirements

1. A client should fetch every asset it needs to draw an avatar, quickly, anywhere in the world.
2. A creator should be able to publish a new version of an item, and players should get it.
3. A moderator should be able to take an asset down, and it should stop being served everywhere within about a minute.
4. It should work for the millions of items almost nobody requests, not only the popular ones.

Below the line (out of scope):

• The authoring and upload pipeline, and the format conversion behind it. I care that ingest produces immutable named blobs; how a mesh gets optimised is a different hour.
• The marketplace, pricing and the economy. Entitlement shows up once, in the API section, because it changes where authorisation lives, and then it goes away.
• Rendering, and the moderation DECISION itself. Classifying an asset as banned is a machine-learning problem; propagating that ban is a delivery problem, and only the second one is mine.

Non-Functional Requirements

• Do the shape of the catalogue first, because it is the opposite of what a CDN expects.
  • 50M creator-made assets, averaging about 300 KB, is roughly 15 TB at origin. That is small. Storage is not the problem here.
  • 20M daily users fetching about 200 DISTINCT assets a session is 4 billion requests a day: about 46,000 a second, call it 139,000 at peak with a 3x diurnal factor that is conservative for a globally spread player base.
  • Say distinct, and say why, because the raw number is much larger. A single crowded scene naively wants 1,500 fetches (100 avatars times 15 assets). Almost all of that collapses: players share default bodies and popular items, and the client already holds most of it from earlier in the session. The 200 figure is the deduplicated working set, and the 1,500 is what you would fetch if you did none of the work in dive 5.
  • At 300 KB that is 1.2 PB a day leaving the edge. So the catalogue is tiny and the traffic is enormous, which tells you immediately that this is a caching problem and not a storage problem.
• Then the distribution, which is what makes UGC different from a normal CDN workload.
  • About ten thousand popular items are most of the traffic and will sit hot in every PoP.
  • The other tens of millions are requested rarely, unpredictably, and from anywhere. A normal CDN assumes a hot working set; user generated content does not have one.
  • That tail is a minority of requests and nearly all of the origin load, which is the number the design has to attack.
• Latency: p99 under about 100 ms, and notice immediately that this constrains the hit rate rather than being independent of it.
  • At a 98 percent hit rate the 98th percentile fetch IS a miss, so p99 is an origin round trip and the target is unreachable. Holding p99 at the edge needs better than 99 percent, realistically 99.5 and up once shield misses are counted.
  • The number that matters to a player is worse than the per-asset one, because an avatar is 15 assets: at 98 percent per asset, 26 percent of avatars contain at least one miss. At 99.5 percent it is 7 percent, at 99.9 percent it is 1.5.
  • So hit rate is a latency argument before it is a cost argument, and that is the strongest case for the shield in the whole design.
• Cache hit rate: target 99.5 percent or better, and be precise about what it moves. The 1.2 PB a day going to clients is fixed by the product; hit rate does not change CLIENT egress. What it changes is origin fill and fleet size (24 TB a day at 98 percent, 6 TB at 99.5, 1.2 TB at 99.9), and the avatar-level miss rate above.
• Takedown propagation under about 60 seconds, globally. This directly contradicts caching for a year, and that contradiction is the most interesting thing on the page rather than an oversight.
• Publish visibility of seconds to a minute is fine. Creators tolerate a short delay; players never notice one.
• Graceful degradation: a failed asset fetch should render a placeholder, not block the scene. Availability here is about never blocking a frame, not about never missing bytes.

Defining the Core Entities

• Asset is a blob of bytes named by their own hash: contentHash, bytes, mime type. It is immutable by construction, because changing the bytes changes the name.
  • The server computes that hash, never the client, and this is a security property rather than a detail. If an uploader can declare the digest, they can store bytes under someone else's hash and poison a year-TTL entry in 200 PoPs, in a system whose whole design has no purge path. Ingest rehashes what it received and rejects a mismatch.
  • This is the decision the whole design rests on, so say it in the entity list rather than saving it for a dive.
• Item is what a creator and a player think of as "a hat": itemId, plus a pointer to the current contentHash and the history of previous ones. The item is mutable; the asset never is.
• Variant is one rendition of an item for one target: a level of detail, a texture format, a platform. Each variant is its own asset with its own hash, because they are genuinely different bytes.
• Composite is a baked low-detail avatar, keyed by a hash of its INPUTS (the sorted component hashes plus LOD and format) because the output hash is unknown until after baking. That input-to-output mapping is the design's second mutable pointer, and it is listed here rather than discovered in a dive.
• Manifest is the mutable pointer: avatar id to a list of slot and content hash. In a session the client is PUSHED this by the session server; the HTTP object below is the cold-entry and out-of-session path.
  • Size it rather than calling it small. Fifteen slots with full URLs and sha256 hashes is about 2 KB. Truncating the hash to 128 bits, still far beyond collision risk here, gets it to roughly 1.5 KB. Bare hashes with no URLs would reach about 600 bytes, but that hands URL construction back to the client, which the API section deliberately refuses. Either way it is cheap in bytes; none of them is 400.
  • It changes whenever the player changes clothes, which is rare per player and constant in aggregate.
• DenyList is the set of banned content hashes. It is small and it is checked on every single request, which makes it a data-structure decision rather than a table. It is a SET rather than an append-only log, because bans get reversed on appeal and un-banning has to be as cheap as banning.

API or System Interface

WS   roster.push                     // session server -> client, the common path
                                     // { avatarId: [ { slot, url } ], ... } for the whole instance

GET  /v1/avatars/{userId}/manifest   // -> { slots: [ { slot, url } ] }, the fallback
                                     // Cache-Control: max-age=30

GET  /assets/{contentHash}.{ext}     // -> the bytes
                                     // Cache-Control: public, max-age=31536000, immutable

POST /v1/items/{itemId}/versions     // creator publishes -> { contentHash }
POST /v1/moderation/deny             // { contentHash } -> propagates to every PoP

• Two objects with deliberately opposite cache policies, and that contrast IS the architecture. One is about 2 KB and changes when a player changes clothes. The other is 300 KB, can never change, and is cached for a year. Giving both the same policy is the mistake the question is hinting at. The small one mostly does not travel over HTTP at all, which is the next step of the same argument.
• No ETag on the asset, because the URL is the version. Conditional requests exist to ask "has this changed", and the answer here is structurally always no.
• The manifest returns URLs, not ids. The client should never construct a URL, so the server can move assets, change variant selection or point at a different CDN without shipping a client.
• Publishing returns a hash, not a success flag. The creator's new version exists the moment the bytes are stored; making it visible is a separate, tiny write to the item pointer.
• Entitlement is enforced at the MANIFEST, never at the CDN, and be careful with the reason, because the obvious one is wrong.
  • Signed URLs do NOT cost you cacheability: CDN token auth deliberately excludes the signature from the cache key, so the object still caches once for everyone entitled.
  • The real reason is that the bytes are not the secret. Ownership is. A cosmetic asset leaks from the client the first time it renders, so signing adds key management and clock skew to protect something that is not protectable.
  • That would change if the asset were genuinely confidential, and saying so is what makes it a decision rather than a habit.

1) Two objects with opposite rules

• There are exactly two kinds of object here, and separating them is most of the design: one small mutable pointer saying which versions are current, and many large immutable assets. How that pointer reaches the client is the next question, and the answer turns out not to be HTTP.
• The manifest is the only thing that ever needs to change quickly. About 2 KB with a 30 second TTL, so it expires on its own and nothing has to be purged.
• Now check that the cheap object is actually cheap, because the whole design rests on it and bytes are the wrong unit.
  • Your own manifest, refreshed on a 30 second TTL across a 30 minute session, is about 14,000 requests a second at 20M daily users. Affordable, but already a third of the asset rate.
  • Other players' manifests are the problem. A scene shows other avatars, and each one is a manifest. At a million concurrent players with 100 visible avatars on a 30 second TTL that is over 3 million requests a second, roughly seventy times the asset traffic the entire design is sized around.
  • And it caches badly, which is the part that turns it from expensive into dangerous. A per-avatar key has about one reader, so a 30 second TTL buys almost no edge hits. Manifest misses, not asset misses, would become the dominant origin load.
  • So the manifest does not travel over HTTP in the common case. The session server already knows the instance roster and already holds a socket to every client, so it pushes the hash list at spawn and pushes a delta when someone changes clothes. Avatar equipment is game state, and shipping game state over a CDN was the mistake.
  • The CDN manifest object stays, for cold entry, for out-of-session views like a profile page, and as the fallback when the push path is unavailable. It is then a small fraction of that rate rather than all of it.
• The assets are cached for a year and served from the nearest PoP. A content hash can never point at different bytes, so there is no staleness to reason about at all.
• Failure is a placeholder, not an error dialogue. A missing asset renders as a default shape; the scene keeps running. That is what lets the whole path be aggressive about caching rather than defensive about correctness.
• The session server belongs in this picture, and it is the component most answers leave out. It already knows who is in the instance and already holds a socket to every client, so it is the natural transport for the roster of hashes and the natural trigger for pre-warming. Treating avatar equipment purely as a CDN object ignores the one service that already has the answer.

2) Content addressing, and refusing the trade-off

• Name the bytes by their hash and the invalidation problem stops existing. A new version is different bytes, so it is a different hash, so it is a different URL. The old URL is still valid and still cached; nobody asks for it any more.
• That is why the stated trade-off is a false one. "Faster delivery versus invalidation complexity" is the tension you get from mutable URLs, where a short TTL is the only way to bound staleness, and a short TTL is exactly what destroys hit rate.
• The mutability does not vanish, it gets confined, and "cheap" has to be earned in the right unit.
  • In bytes it is obviously cheaper: about 2 KB against a 300 KB blob replicated to 200 PoPs.
  • In REQUESTS it is not automatically cheaper at all, because a per-avatar object with a 30 second TTL barely caches. That is why the common-case transport is the session server rather than the CDN, and saying so is the difference between a slogan and a design.
• Say what it costs, because it is not free: every version is stored forever, so storage grows monotonically and needs a collector for versions nothing references any more.

3) The long tail, and tiered caching

• The head takes care of itself. About ten thousand popular items sit hot in every PoP and serve at nearly 100 percent hit rate without anyone doing anything clever.
• The tail is the design problem, and its shape is specific: a cold asset is cold in all 200 PoPs at the same time, so one unlucky item can produce 200 independent origin fetches for exactly the same bytes.
• A regional shield collapses that fan-in, and say the real factor rather than the flattering one. Edges miss to a mid-tier instead of to origin, and each shield fills once for everyone behind it. With a handful of regional shields, 200 fetches becomes one per shield, so roughly 10, not 1. Still a 20x cut, and it is the honest number.
• Then claim the win the sizing actually supports: one rendition each is 15 TB and all twelve renditions are about 180 TB, so the whole catalogue fits inside every shield either way. The shield is not a hot-set cache, it is close to a full replica, and that is what makes the high hit-rate rows reachable rather than aspirational.
• A single global shield is the tempting shortcut and it is wrong twice: it is a single point of failure for all origin fill, and it puts a transcontinental hop in front of a 100 ms budget.
• The fan-in is also worse than 200, which is the detail that shows you have run one. A PoP is many cache nodes, so without consistent hashing inside the PoP you multiply by nodes per PoP. That intra-PoP layer is a real component, not a setting.
• Coalesce concurrent misses at the PoP as well. Ten thousand clients asking for the same cold asset in the same second should produce one fill, not ten thousand, and this is a per-PoP setting rather than an architecture.
• Be precise about what hit rate buys. The 1.2 PB a day going to players is fixed by the product. Hit rate moves origin fill and origin fleet size: 24 TB a day at 98 percent, 6 TB at the 99.5 target, 1.2 TB at 99.9.

4) Takedown, the one true invalidation

• Here is where the immutable design bites back, and it is worth walking into deliberately. Everything above says never invalidate. Moderation says this asset must be gone from 200 edges in under a minute.
• Removing it from the manifest is necessary and not sufficient. It stops new references, but anyone holding the direct URL keeps getting a 200 from their nearest PoP for the rest of the year.
• So the fast path is a REFUSAL, not a purge. A deny list of banned hashes is tiny, propagates to every PoP in seconds, and is checked before serving. Evicting the actual bytes can happen lazily afterwards.
• Both mechanisms are needed and they do different jobs: the manifest stops new references, the deny list stops the URL that already leaked.
• The check has to be cheap, because it runs on every request, and the sizing decides the mechanism.
  • Ban a fraction of a percent of a 50M catalogue and you have hundreds of thousands of hashes. At 32 bytes each that is single-digit megabytes, which fits in memory at every PoP, so use the exact set.
  • A bloom filter is ornamental here: where the exact set fits it buys nothing, and where it would help the confirming lookup becomes a network call on the request path.
  • Name the propagation mechanism rather than saying "a config push". It is an edge key-value store or a config activation, and the products differ enough to matter: some cap at a few megabytes, and some document propagation up to 60 seconds, which spends the whole takedown budget before the decision has even landed.
  • Decide the failure mode, because it is the most consequential policy here. A PoP without the current list either serves and alarms, or refuses and causes an outage. Pick one deliberately and write it down.

1) The trade-off in the question, and why I will not take it

2) The cold tail, which is what UGC actually means

3) The hot drop, which is the opposite failure

4) Takedown against a design that never invalidates

5) Composition, and the request amplification nobody costs

Three more dives, briefly

• Variants, and the combinatorial explosion nobody plans for.
  • One logical item becomes several real assets: levels of detail, texture formats per GPU family, platform-specific meshes. Three LODs times four formats is twelve objects per item.
  • Each variant is its own hash, which keeps immutability intact, but it multiplies both storage and the cold-tail problem by the same factor.
  • So correct the headline number when you say this, or the interviewer will. 50M items at twelve renditions is 600M cacheable objects and roughly 180 TB, not the 15 TB quoted earlier, which was one rendition each. The conclusion survives easily, since 180 TB is still small, but the figure has to move with the claim.
  • So generate variants lazily for the tail and eagerly for the head, and let the manifest decide which variant a given client is told about, because that keeps the decision on the server where it can change.
• Garbage collection, which immutability makes mandatory.
  • Nothing is ever overwritten, so storage only grows: old versions, abandoned uploads, derived variants and baked composites all accumulate.
  • Collect by reference: a blob is deletable when no item version, no manifest and no retention policy points at it, and moderation history has to be treated as a reference or you will delete evidence.
  • The safety bound is the client's retention, not the manifest TTL, and getting that wrong deletes live blobs. A client receives its roster at scene load and holds that working set for the whole session, which is minutes to hours, not 30 seconds. Size the grace window against session length and then some.
  • The race that content addressing creates is on the WRITE side and it is easy to miss. A creator uploads bytes that dedup onto an existing blob, and the collector deletes that blob between the dedup hit and the pointer write, so the publish points at nothing. Take a lease or a reference at dedup time, or use a mark epoch that aborts if any new reference appeared while marking.
  • The client needs a collector too, for the same reason the server does. Hash-named assets never expire, so a game client accumulates them forever unless it has a disk budget and an eviction policy. Immutability makes client-side GC mandatory, not optional.
  • Deletion must be slow and reversible, because content addressing means a mistaken delete cannot be repaired by re-uploading something similar. Only the identical bytes restore the URL.
• Cost, which for a delivery system is a first-class design input rather than an afterthought.
  • 1.2 PB a day of client egress is the dominant line and it is fixed by the product, so the levers are compression, level of detail and bundle size, not caching.
  • Caching buys origin fleet and fill, which is the smaller number, and it is worth being honest that hit rate is a capacity story more than a bill story here.
  • Price the requests, not only the bytes. 4 billion requests a day at typical CDN request pricing is on the order of a million dollars a year, roughly half the egress bill, and it is driven entirely by the request amplification in dive 5. Bundling and baking are the only things that move it, which is the cost argument for them rather than the latency one.
  • Cash in the indirection you already built. Because the manifest hands out URLs rather than ids, steering a fraction of traffic to a second CDN is a server-side change with no client release. On a design betting 1.2 PB a day on one vendor, that is the cheapest availability insurance going, and it is already paid for.
  • Residency and erasure collide with immutability, and this is the sharpest legal edge here. "Stored forever" and "cached for a year in 200 PoPs" sit against an erasure obligation on user-generated content, and an EU creator's asset is replicated globally by default. The deny list is most of the enforcement; what is missing is a residency policy on where blobs may be filled from, and an erasure path that actually deletes rather than refuses.
  • The genuinely large saving on bytes is making them smaller: Brotli for meshes and JSON, GPU formats like ASTC or BCn that are already compressed and gain nothing from another pass, and geometry compression. Applied to the head of the distribution, that beats any caching change, because the head is most of the traffic.
  • Size the fleet at PEAK, not at the average. 139,000 a second is the number the origin, the shields and the manifest path have to survive, and quoting an average and then sizing nothing against the peak is a common way to be wrong by 3x.

Final Design

• In a session the client gets its roster of content hashes pushed by the session server; out of session it fetches a manifest of about 2 KB with a 30 second TTL. Either way it then fetches every asset it names by content hash, cached for a year at the nearest of 200 PoPs.
• Assets are named by the hash of their bytes, so publishing a new version creates a new URL and there is no purge path anywhere in the system.
• Regional shields sit between the edges and the origin, so a cold asset costs one origin fill per shield, roughly 10 rather than 200, and each PoP coalesces concurrent misses into a single upstream request.
• Ingest hashes the bytes, builds the variants each platform will ask for, and publishing is a pointer write from the item to the new hash.
• Takedown publishes a hash to a small deny list that every PoP and the origin check before serving, because the manifest alone cannot stop a URL that already leaked.
• A collector reclaims blobs that no item version, manifest or retention rule references, because nothing is ever overwritten.
• The invariant worth closing on: the large objects are immutable and the small object is not, which is what turns the question's trade-off into a non-question. The two things it does not solve are the cold tail and the takedown, and those are the dives worth having.

What is Expected at Each Level

• Mid
  • Puts a CDN in front of an object store and serves assets from the edge.
  • Knows that versioning the URL avoids stale content.
  • Usually accepts the stated trade-off and designs a purge path for it.
  • Usually assumes a hot working set and does not notice the tail.
• Senior
  • Uses content-addressed immutable URLs and long TTLs deliberately, and separates the manifest from the assets.
  • Identifies the cold tail and adds a shield or a mid-tier, and knows what request coalescing is for.
  • Handles takedown as a separate mechanism rather than assuming purge covers it.
  • Sizes the traffic and the catalogue instead of describing them qualitatively.
• Staff
  • Refuses the trade-off in the question explicitly, and can explain that it is an artefact of mutable URLs rather than a law.
  • Says where the mutability went instead of claiming it disappeared, and gives the small object its own policy and path.
  • Separates the two cache-miss problems: many cold assets each asked for rarely, and one cold asset asked for by everyone at a drop.
  • Is precise that hit rate moves origin fill and fleet size, not client egress, rather than overclaiming a cost saving.
  • Treats takedown as the one true invalidation and designs refusal rather than purge, including the origin refill path.
  • Raises composition and request amplification unprompted, and reaches server-side baking for the far case.
  • Names what immutability costs: monotonic storage, a mandatory collector, an irreversible delete, and a leaked URL that lives forever.