Scope: self-managed Cassandra, with capacity chosen explicitly. The node count covers the illustrative write workload. It does not prove that years of history or the full Messenger workload fit.
Start with the workload
For Messenger, assume one million new messages per second, each containing 1 KB of text and metadata. This is a write workload, not a mixture of reads and writes.
| Assumption | Value |
|---|---|
| Arrival rate | 1M new messages/s, sustained |
| Stored payload per message | 1 KB, decimal units |
| Replication factor | 3 |
| History retention | Years, according to product requirements |
| Attachments | Object storage; message rows contain references |
| Main read | Latest messages in a conversation, then paginate backward |
Peak and average are equal in this example. A queue can absorb bursts, but it does not reduce sustained writes or retained bytes. Each message remains a distinct record.
Estimate capacity before provisioning
Memorize a clear assumption, not a supposed universal Cassandra rate:
“I'll assume an SSD-backed node can sustain 20K replica writes per second for this schema, including normal background work, then target 70% utilization.”
1M logical writes/s × RF 3 = 3M replica writes/s 3M ÷ 20K = 150 nodes before headroom 150 ÷ 0.70 ≈ 214.3 → 215 nodes Round to 216 nodes across three AZs
20K is an interview placeholder, not a measured or guaranteed rate. Validate it with the actual schema, consistency, durability, latency target, and maintenance workload. If a benchmark already includes RF 3 in its client-throughput figure, do not multiply by three again.
Storage is a separate constraint
1M messages/s × 1 KB = 1 GB/s 1 day: 86.4 TB original payload 1 year of 365 days: 31.536 PB original payload With RF 3: 94.608 PB, before compression and overhead
Choose storage per node, measure compression, and account for metadata, tombstones, compaction space, and repair. The larger requirement from throughput and storage determines fleet size. Additional query-specific tables and read traffic increase the work.
216 is only the initial write-throughput estimate. Years of history may require far more storage. A low-latency historical tier versus an archive is a product decision, not an automatic 30-day cutoff.
Provision and configure the machines
A node is a machine running a Cassandra server process. In a typical self-managed deployment, each VM or physical server runs one node. Provision the fleet determined by both throughput and storage.
Infrastructure automation creates the machines and storage, installs Cassandra, and supplies each node's configuration.
| Configuration | Purpose |
|---|---|
| Cluster name | For example, messenger-production |
| Private network addresses | How nodes and clients reach this node |
| Seed addresses | Selected Cassandra nodes used for discovery |
| Datacenter | For example, us_east |
| Rack | Use the node’s AZ for replica placement |
| Token allocation | Assign ownership of ranges in the hash space |
Where do the first seeds come from?
Provision the initial machines, obtain their private addresses or DNS names, and designate some as seeds. Configure those addresses before starting the initial cluster. There is no pre-existing master to find.
Seeds are ordinary Cassandra nodes that help with discovery. They are not masters or traffic proxies. Their count is independent of RF: RF 3 does not require three seeds.
Start the cluster and verify membership
Start the initial nodes and join the remaining nodes in controlled batches. The Cassandra process on a joining node discovers the cluster and, when joining a populated cluster, requests its assigned data from existing replicas.
- The joining node acquires token assignments and determines the ranges it must store.
- It requests the corresponding data from existing replica nodes.
- Those nodes stream the data to it while Cassandra handles ongoing writes.
- After bootstrap completes, it becomes a normal serving member.
For this Cassandra 5.x model, there is no dedicated master coordinating every transfer. The joining node coordinates its bootstrap with peers. Seeds provide discovery. Your automation monitors the rollout and streaming.
After expansion, verify the new node and run cleanup on affected old nodes to reclaim data they no longer own. Bootstrap consumes disk and network capacity, so adding a node is not instantaneous relief.
Create the keyspace first
A keyspace groups tables and owns their replication policy. It resembles a namespace, but also configures where copies of the data live.
01-keyspace.cql · CQL
CREATE KEYSPACE messenger
WITH replication = {
'class': 'NetworkTopologyStrategy',
'us_east': 3
};The datacenter name must match the node configuration. RF 3 means three complete replicas of every partition in that datacenter. With correct AZ/rack configuration, topology-aware placement spreads them across three AZs.
Create the keyspace before creating its tables. Tables inherit replication from their keyspace; there is no separate RF setting on each table.
Create the table around the query
This table serves conversation history, newest first. The application supplies the daily bucket value.
02-messages.cql · CQL
CREATE TABLE messenger.messages_by_conversation (
conversation_id uuid,
bucket date,
message_id timeuuid,
sender_id uuid,
body text,
PRIMARY KEY ((conversation_id, bucket), message_id)
) WITH CLUSTERING ORDER BY (message_id DESC);| Schema element | Meaning |
|---|---|
| messenger | Keyspace |
| messages_by_conversation | Table |
| (conversation_id, bucket) | Composite partition key |
| message_id | Clustering column and identifier within the partition |
| DESC | Newest-first clustering order within each partition |
No TTL is configured, so messages do not automatically expire. Cassandra stores the whole partition at each of its three replica nodes.
What does bucketing do?
Rather than one conversation partition growing forever, each date identifies another partition:
(chat_123, 2026-09-29) → one partition (chat_123, 2026-09-30) → another partition
Daily buckets are illustrative. Busy conversations may need shorter intervals or size-based buckets. A time bucket bounds history growth; it does not spread all current writes for one conversation. Additional sharding addresses that problem, at the cost of merging reads.
Read primary-key syntax
Primary key = partition key + optional clustering columns. The first entry is the partition key. Inner parentheses combine columns into that entry.
| Declaration | Partition key | Clustering columns |
|---|---|---|
PRIMARY KEY (a) | a | None |
PRIMARY KEY (a, b) | a | b |
PRIMARY KEY (a, b, c) | a | b, c |
PRIMARY KEY ((a, b), c) | a and b together | c |
There is no separate partition_key = (a, b) declaration. The parentheses inside PRIMARY KEY explicitly define it.
Using only message_id as the primary key is valid. Each message then has its own partition, which suits lookup by ID but does not directly serve conversation-history queries.
Clustering order is optional
Ascending order is the default. DESC makes newest-first the default for this table. Cassandra can also query in reverse clustering order. Ordering applies within a partition, not globally across the table.
Other examples: newest-first transactions within a user/month partition, or oldest-first sensor readings within a sensor/day partition.
How deep can names go?
Qualified table names have two levels: keyspace.table. Keyspaces cannot nest. The data hierarchy continues as cluster, keyspace, table, partitions, rows, columns; those are not extra namespace levels.
Understand nodes, tokens, and partitions
A node is a machine. A token is a marker in the hash space. A partition is a group of rows sharing a partition key.
| Concept | What it means |
|---|---|
| Physical node | Runs Cassandra and stores many partitions |
| Token / vnode | A token marks ownership of a hash range; a node usually owns multiple ranges |
| Partition | Rows with one partition-key value, copied as a whole to replica nodes |
| Table | Schema and data; its partitions can span many nodes |
Hash(conversation_id, bucket) → position in the token space → primary range owner → additional replicas selected by the keyspace policy
The ring is a logical map, not a network chain. Requests do not travel node by node around it.
A node's ranges can contain partitions from many tables. Adding a table does not require new vnodes. Table identities keep data separate, even when keys hash to the same position.
One conversation today, three full copies
| Partition | Illustrative replica nodes |
|---|---|
| (chat_123, today) | A, B, C |
| (chat_456, today) | B, D, E |
| (chat_123, tomorrow) | A, D, F |
These node letters are examples, not computed placements. All today's messages for chat_123 are in one partition, with a full copy on A, B, and C. Tomorrow's bucket may map elsewhere. Using only the date as a partition key would concentrate the whole day's traffic into one partition.
Multiple tables share the same cluster resources. You do not create a table per conversation; you create many conversation partitions in the same table.
Connect the application
Give the Cassandra driver contact points and the local datacenter. Contact points are client discovery addresses; seed addresses are node discovery configuration.
Contact points: node-A, node-B, node-C Local datacenter: us_east Write consistency: LOCAL_QUORUM Read consistency: LOCAL_QUORUM Request execution: bounded asynchronous concurrency
The driver discovers membership and token ownership. With partition-key information, commonly from prepared statements, token-aware routing prefers a replica for that partition.
Any node can coordinate a request. A non-replica coordinator contacts the replicas, which may add a network hop. “Coordinator” describes a role for that request, not a permanent master.
With RF 3, LOCAL_QUORUM needs responses from two replicas. Writes still go to all three. Configure commit-log synchronization to match the promised durability: quorum acknowledgments alone do not mean every copy was synchronously flushed to disk.
Serve writes and reads
Write path
- The message service generates a stable message ID and bucket.
- The driver routes the write to a coordinator.
- The coordinator sends it to replicas.
- The requested acknowledgments complete the database operation.
- The delivery system notifies recipients.
Retries reuse the same message ID and bucket to target the same row. This does not automatically make notifications or other side effects exactly once.
Read path
- Identify the conversation's latest relevant bucket.
- Query a page of messages in clustering order.
- Continue within that bucket, then into older buckets.
Keep enough bucket metadata or use a deterministic bucket scheme so the application can find older history efficiently. Cache recent history when repeated reads justify it.
If there is a queue first
Define whether “sent” means durably accepted into Kafka or persisted in Cassandra. Asynchronous persistence requires a plan for read-after-send behavior. Consumers must still sustain the long-term arrival rate.
Batching can reduce request overhead without reducing records or bytes. Large cross-partition Cassandra batches are not a general throughput optimization; bounded concurrent writes are often the appropriate mechanism.
Choose a lever based on the bottleneck
| Observed constraint | Lever | Tradeoff |
|---|---|---|
| Cluster-wide saturation | Add or upgrade nodes | Bootstrap and maintenance need spare capacity |
| Hot conversation | Shard current writes | Reads merge across shards |
| Unbounded partition growth | Smaller history buckets | Reads may cross more buckets |
| Repeated history reads | Cache recent messages | Cache consistency and invalidation |
| Expensive reads | Query-specific table design | Extra writes and storage |
| Secondary write load | Coalesce updates where semantics permit | Potential delay in derived state |
| Historical storage | Compression, more storage, agreed archival tier | CPU, recovery time, historical latency |
| Background disk pressure | Tune compaction and repair | Workload-specific testing required |
RF and consistency also affect cost and latency, but they change guarantees. Waiting for fewer write acknowledgments does not reduce the configured number of replicas.
Observe p95/p99 latency, errors, dropped requests, CPU, disk usage and latency, compaction backlog, partition skew, and repair health. Test failure capacity separately from normal-operation headroom.
The interview answer
“I'd partition message history by conversation and a bounded bucket, and use RF 3 across three AZs. At an assumed 20K replica writes per second per node, one million writes per second needs about 216 nodes with headroom. I'd separately size retained history and reads. I'd create the cluster, then the RF-3 keyspace, then the tables, and connect token-aware clients. Hot conversations need sharding; broad saturation needs more capacity.”
Workload and queries → capacity → provision nodes → start cluster → create keyspace → create tables → connect clients → validate and scale.
Sources and scope
This guide describes a Cassandra 5.x-style deployment. Exact configuration and topology procedures depend on the installed version. The 20K rate is an illustrative assumption, not a published benchmark or a claim that this design has been load-tested.