← all topics

System design / Worked example

Kafka sizing,
from traffic to storage.

Size MSK Express using published throughput, then calculate retention costs and consumer parallelism.Size brokers with fixed local storage for throughput, retention, and consumer parallelism.

On: managed storage and published Express throughput.

Combined producer traffic
1.2 GB/s
Two topics, three copies
MSK Express sizing estimate
15 brokers
express.m7g.4xlarge, five per AZ
Express / one-day retention
~$73.9k
USD per 30-day month, before excluded costs
Fixed storage / seven-day retention
390 brokers
8 TB each, 130 per AZ
Chosen logical partitions
780
650 for A, 130 for B

MSK Express is on. This view uses 15 brokers and one-day retention, with a seven-day cost comparison. Storage is managed separately.

MSK Express is off. This view uses the generic fixed-storage assumptions and seven-day retention.

Step 01

Define the workload

AssumptionTopic ATopic B
Events per second1,000,000200,000
Average event size1 KB1 KB
Incoming data1,000 MB/s200 MB/s
Retention1 day7 days1 day7 days
Replication factor33
Groups reading the full topic11

Combined incoming traffic is 1,200 MB/s. Use decimal units throughout: 1 KB = 1,000 bytes; 1 TB = 1,000 GB. Uppercase B means bytes.

Assume this rate runs continuously and ignore compression savings. In a real estimate, use peak traffic for throughput and average traffic for storage. This example makes average equal to peak.

Replication factor 3 means three stored copies of each logical partition: one leader and two followers. Kafka calls all three copies replicas. A producer writes to the leader; each follower receives the same data.

Step 02

Choose fixed-storage broker assumptions

These are explicit interview assumptions to validate with benchmarks. They are not universal Kafka specifications.

ResourceAssumption per broker
Aggregate disk-write capacity100 MB/s
Network inbound capacity1,000 MB/s
Network outbound capacity, simultaneously1,000 MB/s
Disk storage capacity8 TB
Target utilization / occupancy70%

The network budget represents roughly usable throughput on a 10 Gbps connection. All leader and follower replicas on a broker share its disk and network resources.

MB/s measures the rate of work. TB measures the amount stored. Check both limits separately.

Step 03

Calculate cluster throughput demand

Disk writes
1,200 MB/s × 3 copies = 3,600 MB/s

Network inbound
1,200 MB/s from producers
+ 2,400 MB/s received by followers
= 3,600 MB/s

Network outbound
2,400 MB/s sent to followers
+ 1,200 MB/s sent to consumers
= 3,600 MB/s

Replication appears in both network directions because one broker sends what another receives. We compare each direction against its own capacity. The calculation omits protocol overhead.

More consumers within the same group divide its processing work. An additional group reading the full stream adds another stream's worth of consumer traffic.

Step 04

Size brokers for throughput

Apply the 70% utilization target to each resource.

ConstraintCalculationBrokers, rounded up
Disk writes3,600 ÷ (100 × 0.70)52
Network inbound3,600 ÷ (1,000 × 0.70)6
Network outbound3,600 ÷ (1,000 × 0.70)6

Take the largest requirement, then round for equal placement across three availability zones.

54 brokers total, or 18 per AZ, satisfy the throughput estimate.

Do not add these counts. The same machines handle disk writes and network traffic. Before headroom, the disk-write minimum was 3,600 ÷ 100 = 36 brokers.

Step 05

Size brokers for seven-day retention

7 days = 604,800 seconds

Data before replication
1.2 GB/s × 604,800 = 725,760 GB = 725.76 TB

Data with three copies
725.76 TB × 3 = 2,177.28 TB

Storage budget per broker
8 TB × 0.70 = 5.6 TB

Brokers needed
2,177.28 ÷ 5.6 = 388.8

Round up and distribute evenly across three AZs: 390 brokers total, or 130 per AZ.

RequirementBroker count
Throughput with headroom54
Seven-day storage with headroom390
Choose the larger count390

Storage drives this result. Longer retention increases historical data stored; it does not increase the incoming write rate.

These are payload estimates. Indexes, segment deletion timing, and operational overhead still need allowance. The 70% occupancy target leaves physical space, but the result is close to that chosen target.

Step 06

Determine consumer parallelism

Assume one consumer can process 10,000 events per second for either topic's workload.

Topic A: 1,000,000 ÷ 10,000 = 100 consumers minimum
Topic B:   200,000 ÷ 10,000 =  20 consumers minimum

In a standard consumer group, each partition is assigned to one consumer at a time. Supporting those consumers therefore requires at least 100 partitions for A and 20 for B.

These are processing minimums without consumer headroom. Partition count also needs to support storage distribution across the broker fleet.

Step 07

Choose partitions for storage placement

For the 390-broker fixed-storage example, choose 780 logical partitions: two leaders per broker on average. Split them by the topics' 5:1 traffic ratio: 780 × 5/6 = 650 for A, and 780 × 1/6 = 130 for B. This is a convenient choice, not a calculated minimum.

CountTopic ATopic BTotal
Logical partitions650130780
Stored copies, RF 31,9503902,340
Incoming traffic per logical partition
1,200 MB/s ÷ 780 ≈ 1.54 MB/s

Seven-day data per stored copy
725.76 TB ÷ 780 ≈ 0.93 TB

Stored copies per broker
2,340 ÷ 390 = 6

Storage per broker
2,177.28 TB ÷ 390 ≈ 5.58 TB

With balanced keys and placement, each broker hosts five copies from Topic A and one from Topic B. That is two leaders and four followers on average. All six copies count toward storage: 2 × 0.93 TB + 4 × 0.93 TB ≈ 5.58 TB. Each leader's followers live on other brokers; the local followers belong to other partitions.

A replica resides on one broker. Partition sizes must fit the storage budget of individual brokers; dividing total TB by broker count alone is insufficient.

780 partitions does not require 780 consumers. A consumer can read multiple partitions. Choose consumer count from processing demand and headroom.

Kafka requires neither power-of-two partition counts nor exact multiples of broker count. These values make this worked example evenly divisible. Actual per-partition throughput still needs a benchmark.

Step 08

Verify the fixed-storage layout

Per-broker checkEstimated loadTarget budget
Disk writes9.23 MB/s70 MB/s
Network inbound9.23 MB/s700 MB/s
Network outbound9.23 MB/s700 MB/s
Retained data5.58 TB5.6 TB

The throughput checks divide 3,600 MB/s by 390 brokers. The storage check divides 2,177.28 TB by 390. These are averages under balanced placement.

One cluster across three availability zones

AZ A130 brokers
AZ B130 brokers
AZ C130 brokers

Configure AZ-aware replica placement so every partition has one copy in each AZ. Its leader can be in any AZ, with leaders balanced across the cluster. 390 is the total count across all three AZs. Replication is already included in the sizing.

This verifies steady-state budgets under the assumptions. Failure recovery, historical replay, CPU, data skew, and actual partition performance remain separate checks. A 70% target by itself does not prove survival of an AZ outage.

Step 09

Evaluate the deployment choices

The result is 390 brokers because this example fixes storage at 8 TB each and retains all seven days on broker storage. Larger storage or tiered storage could reduce the broker count.

Keeping 54 brokers would require about 57.6 TB provisioned per broker at 70% occupancy. That is a generic hardware alternative, not an available MSK Standard configuration.

AWS optionStorage implication
MSK StandardUp to 16 TiB of broker storage. A 390-broker cluster also exceeds the default cluster quota.
MSK Standard with tiered storageSeparate local retention from total retention; size local and remote storage separately.
MSK ExpressStorage scales automatically; no per-broker disk size to provision.
Kafka managed on EC2Storage depends on the instance and attached volumes; validate bandwidth and recovery time.

AWS capacities and quotas can change. Consult the linked service documentation before selecting a deployment. TB in the calculations is decimal; AWS's TiB limit is binary.

Step 10 / Managed storage alternative

Size Amazon MSK Express

Use 1,200 MB/s of producer traffic, one full-stream consumer group per topic, and three copies. Size against AWS's published service throughput for express.m7g.4xlarge.

Published sustained capacityPer broker
Producer ingress124.9 MB/s
Consumer egress249.8 MB/s

Producer ingress is the supported rate for accepting producer data, with allowance for storage, replication, and background operations. It is not a raw disk-write or network-interface specification. AWS manages the underlying storage; consumer egress remains a separate check. AWS Express sizing guidance.

Use our 70% planning target on the published sustained rates

Ingress: 1,200 ÷ (124.9 × 0.70) ≈ 13.73 → 14 brokers
Egress:  1,200 ÷ (249.8 × 0.70) ≈  6.86 →  7 brokers

Take the larger count and round for three AZs
15 brokers total = 5 per AZ

Do not multiply the 1,200 MB/s by replication factor again. The Express service limits already account for the work behind producer ingress.

Verify per-broker client traffic

Producer ingress: 1,200 ÷ 15 = 80 MB/s
Planning budget: 124.9 × 70% = 87.43 MB/s

Consumer egress: 1,200 ÷ 15 = 80 MB/s
Planning budget: 249.8 × 70% = 174.86 MB/s

Both fit under balanced traffic. This is a capacity estimate, not an executed workload benchmark or proof of outage recovery.

Retention affects storage cost, not this broker count

Express storage scales automatically. One-day and seven-day retention both use the same 15-broker throughput estimate. We do not reserve 8 TB disks or divide retained bytes by a per-broker disk size. Express storage documentation.

Revisit partitions independently

Using our assumed 10,000 events/s per consumer, the minimum is 100 partitions for A and 20 for B. An illustrative choice is 135 for A and 27 for B, with additional room for consumer parallelism. More partitions remain valid if the workload needs them.

At 162 logical partitions and RF 3, there are 486 copies, averaging 32.4 per broker across 15 brokers. Exact divisibility is not required. Validate partition traffic, key balance, processing headroom, and service limits.

Step 11 / Cost estimate

Price MSK ExpressPrice the Standard baseline

Use US East (N. Virginia), USD on-demand prices, and a 30-day month of 720 hours. Assume 1.2 GB/s continuously, no compression, and a full retention window already accumulated. These estimates exclude client transfer, producer and consumer compute, monitoring, PrivateLink, taxes, and discounts.

Earlier sizing uses decimal GB and TB. The calculations below convert actual bytes to GiB for AWS's billing convention. This changes the unit labels, not the workload.

Monthly ingestion
1.2 × 10⁹ bytes/s × 86,400 × 30 ÷ 2³⁰
≈ 2,896,786 GiB

One-day retained logical data
1.2 × 10⁹ bytes/s × 86,400 ÷ 2³⁰
≈ 96,560 GiB

Seven-day retained logical data
96,559.52 × 7 ≈ 675,917 GiB

MSK Express: about $73,875/month for one day

ChargeRate / basis1 day retained7 days retained
15 Express brokers15 × 720 × $3.264/hour$35,251$35,251
Ingestion$0.01 per billing GB$28,968$28,968
Retained logical storage$0.10 per billing GB-month$9,656$67,592
Monthly subtotalRounded to nearest dollar$73,875$131,811

Express bills retained logical data once. Do not multiply its storage charge by RF 3. Reducing retention changes the storage bill; it does not change the incoming traffic or the broker estimate. MSK does not charge internal broker replication transfer, but client data transfer may incur charges.

Standard baseline: about $347,872/month for seven days

To price the earlier generic fixed-storage example, assume 390 kafka.m7g.large brokers with 7,451 GiB each, approximately 8 decimal TB. The instance choice needs workload validation and this fleet requires quota approval.

ChargeCalculationMonthly cost
Brokers390 × 720 × $0.204/hour$57,283
Provisioned storage390 × 7,451 GiB × $0.10/month$290,589
SubtotalSeven-day fixed-storage baseline$347,872

This Standard estimate includes provisioned space for all replicas and unused capacity. It is not an optimized Standard deployment. Standard with tiered storage needs its own sizing and cost comparison before choosing a service.

Rates were checked against AWS's US East price list, published September 11, 2026. AWS labels these usage rates GB and GB-month; the byte conversion here follows its GiB-based sizing convention. Actual bills include metadata and other measured usage. An initially empty cluster has lower storage usage while retention fills.

AWS MSK pricing · AWS regional price list · AWS billing and sizing worked examples

Step 10

Add brokers to a running cluster

Adding the broker is the easy part. Kafka does not rebalance on its own, so the work is moving data onto it.

1. Start the broker

Give it a unique node.id and point it at the same cluster (in KRaft mode, the same controller quorum). It registers and appears in the cluster metadata, holding nothing. Existing partitions stay where they are.

2. Only new data lands on it for free

New topics, and new partitions of existing topics, can be placed on the new broker. Old partitions never move by themselves: a cluster you grow still carries all its load on the original brokers.

3. Move partitions with a reassignment

kafka-reassign-partitions.sh --generate    # propose a plan for the chosen topics
kafka-reassign-partitions.sh --execute --reassignment-json-file plan.json --throttle 50000000
kafka-reassign-partitions.sh --verify      # confirm it finished, then remove the throttle

The new broker joins as an extra follower and copies the partition's whole log from the leader. Once it has caught up and is in the in-sync replica set, it replaces the old replica, which is dropped. Producers and consumers keep working throughout.

4. Throttle the copy

Moving a replica copies all of its retained data over the network and onto the new disk. Without a throttle, that traffic competes with producers and consumers and latency rises.

5. Rebalance leaders

After replicas move, run a preferred leader election (or leave auto.leader.rebalance.enable on) so leadership, and with it client traffic, spreads onto the new brokers too.

Worked example: 390 to 420 brokers

Add 30 brokers, 10 per AZ, so placement stays even across the three AZs.

Data per broker after the move
2,177.28 TB ÷ 420 ≈ 5.18 TB

Data copied onto the 30 new brokers
30 × 5.18 TB ≈ 155.5 TB, about 0.4 TB sent by each existing broker

Throttle: the new broker's disk-write budget from Step 02
100 MB/s × 0.70 = 70 MB/s, so a 50 MB/s throttle leaves room for live writes

Time to fill one new broker
5.18 TB ÷ 50 MB/s ≈ 103,700 s ≈ 29 hours

Growing this cluster takes more than a day of copying, run in parallel across the 30 new brokers.

The 2,340 stored copies do not divide evenly by 420: 240 brokers hold 6 copies and 180 hold 5. Partitions are the unit of movement, so balance is only as fine as the partition count allows.

With MSK Express

Express keeps broker storage managed and separate from the broker count, so a reassignment has far less local log to copy than the fixed-storage example. The broker count still comes from throughput: adding one broker per AZ takes this example from 15 to 18 and lowers per-broker client traffic from 80 MB/s to about 67 MB/s. Check the MSK documentation for how Express rebalances partitions onto new brokers.

In practice

OptionHow rebalancing works
Cruise ControlLinkedIn's open-source tool computes and runs the reassignment plan from disk, network and CPU load.
MSK StandardRaise the broker count in the console, in multiples of the AZ count, then rebalance with Cruise Control or the reassignment tool.
MSK ExpressStorage is not tied to the broker, so there is far less data to move.
Tiered storageOnly the local, hot part of each log moves, which cuts reassignment time sharply.

What interviewers listen for. Partitions are the unit of movement: you can move one but never split one, so choose enough partitions up front for the brokers you will eventually run. Adding partitions later is a different operation, and keys then hash to new partitions, which breaks per-key ordering across the change. Scaling out is not instant: new brokers help only after data is copied onto them, so add capacity ahead of a peak, not during one.

Use in the interview

Explain the deployment choice

"The topics produce 1.2 GB/s combined. Using AWS's published throughput for express.m7g.4xlarge, I estimate 15 brokers with headroom across three AZs. Storage scales separately. At sustained traffic, one-day retention costs about $74,000 per month, versus $132,000 for seven days, before client transfer and application costs. I'd validate throughput and size consumers and partitions for the processing workload."
"The topics produce 1.2 GB/s combined. Under our assumed disk and network capacities, throughput needs about 54 brokers with headroom. Seven-day retention raises the requirement to 390 brokers with 8 TB each. I'd choose partitions for processing and storage placement, then evaluate larger disks or tiered storage."

Workload → deployment model → throughput capacity → retention and billing → partitions and consumers → placement and failure checks.

Sources and scope

The fixed-storage hardware and consumer processing rates are illustrative assumptions. Express throughput and AWS prices use the linked published figures. No workload benchmark was performed.