MSK Express is on. This view uses 15 brokers and one-day retention, with a seven-day cost comparison. Storage is managed separately.
MSK Express is off. This view uses the generic fixed-storage assumptions and seven-day retention.
Define the workload
| Assumption | Topic A | Topic B |
|---|---|---|
| Events per second | 1,000,000 | 200,000 |
| Average event size | 1 KB | 1 KB |
| Incoming data | 1,000 MB/s | 200 MB/s |
| Retention | 1 day7 days | 1 day7 days |
| Replication factor | 3 | 3 |
| Groups reading the full topic | 1 | 1 |
Combined incoming traffic is 1,200 MB/s. Use decimal units throughout: 1 KB = 1,000 bytes; 1 TB = 1,000 GB. Uppercase B means bytes.
Assume this rate runs continuously and ignore compression savings. In a real estimate, use peak traffic for throughput and average traffic for storage. This example makes average equal to peak.
Replication factor 3 means three stored copies of each logical partition: one leader and two followers. Kafka calls all three copies replicas. A producer writes to the leader; each follower receives the same data.
Choose fixed-storage broker assumptions
These are explicit interview assumptions to validate with benchmarks. They are not universal Kafka specifications.
| Resource | Assumption per broker |
|---|---|
| Aggregate disk-write capacity | 100 MB/s |
| Network inbound capacity | 1,000 MB/s |
| Network outbound capacity, simultaneously | 1,000 MB/s |
| Disk storage capacity | 8 TB |
| Target utilization / occupancy | 70% |
The network budget represents roughly usable throughput on a 10 Gbps connection. All leader and follower replicas on a broker share its disk and network resources.
MB/s measures the rate of work. TB measures the amount stored. Check both limits separately.
Calculate cluster throughput demand
Disk writes 1,200 MB/s × 3 copies = 3,600 MB/s Network inbound 1,200 MB/s from producers + 2,400 MB/s received by followers = 3,600 MB/s Network outbound 2,400 MB/s sent to followers + 1,200 MB/s sent to consumers = 3,600 MB/s
Replication appears in both network directions because one broker sends what another receives. We compare each direction against its own capacity. The calculation omits protocol overhead.
More consumers within the same group divide its processing work. An additional group reading the full stream adds another stream's worth of consumer traffic.
Size brokers for throughput
Apply the 70% utilization target to each resource.
| Constraint | Calculation | Brokers, rounded up |
|---|---|---|
| Disk writes | 3,600 ÷ (100 × 0.70) | 52 |
| Network inbound | 3,600 ÷ (1,000 × 0.70) | 6 |
| Network outbound | 3,600 ÷ (1,000 × 0.70) | 6 |
Take the largest requirement, then round for equal placement across three availability zones.
54 brokers total, or 18 per AZ, satisfy the throughput estimate.
Do not add these counts. The same machines handle disk writes and network traffic. Before headroom, the disk-write minimum was 3,600 ÷ 100 = 36 brokers.
Size brokers for seven-day retention
7 days = 604,800 seconds Data before replication 1.2 GB/s × 604,800 = 725,760 GB = 725.76 TB Data with three copies 725.76 TB × 3 = 2,177.28 TB Storage budget per broker 8 TB × 0.70 = 5.6 TB Brokers needed 2,177.28 ÷ 5.6 = 388.8
Round up and distribute evenly across three AZs: 390 brokers total, or 130 per AZ.
| Requirement | Broker count |
|---|---|
| Throughput with headroom | 54 |
| Seven-day storage with headroom | 390 |
| Choose the larger count | 390 |
Storage drives this result. Longer retention increases historical data stored; it does not increase the incoming write rate.
These are payload estimates. Indexes, segment deletion timing, and operational overhead still need allowance. The 70% occupancy target leaves physical space, but the result is close to that chosen target.
Determine consumer parallelism
Assume one consumer can process 10,000 events per second for either topic's workload.
Topic A: 1,000,000 ÷ 10,000 = 100 consumers minimum Topic B: 200,000 ÷ 10,000 = 20 consumers minimum
In a standard consumer group, each partition is assigned to one consumer at a time. Supporting those consumers therefore requires at least 100 partitions for A and 20 for B.
These are processing minimums without consumer headroom. Partition count also needs to support storage distribution across the broker fleet.
Choose partitions for storage placement
For the 390-broker fixed-storage example, choose 780 logical partitions: two leaders per broker on average. Split them by the topics' 5:1 traffic ratio: 780 × 5/6 = 650 for A, and 780 × 1/6 = 130 for B. This is a convenient choice, not a calculated minimum.
| Count | Topic A | Topic B | Total |
|---|---|---|---|
| Logical partitions | 650 | 130 | 780 |
| Stored copies, RF 3 | 1,950 | 390 | 2,340 |
Incoming traffic per logical partition 1,200 MB/s ÷ 780 ≈ 1.54 MB/s Seven-day data per stored copy 725.76 TB ÷ 780 ≈ 0.93 TB Stored copies per broker 2,340 ÷ 390 = 6 Storage per broker 2,177.28 TB ÷ 390 ≈ 5.58 TB
With balanced keys and placement, each broker hosts five copies from Topic A and one from Topic B. That is two leaders and four followers on average. All six copies count toward storage: 2 × 0.93 TB + 4 × 0.93 TB ≈ 5.58 TB. Each leader's followers live on other brokers; the local followers belong to other partitions.
A replica resides on one broker. Partition sizes must fit the storage budget of individual brokers; dividing total TB by broker count alone is insufficient.
780 partitions does not require 780 consumers. A consumer can read multiple partitions. Choose consumer count from processing demand and headroom.
Kafka requires neither power-of-two partition counts nor exact multiples of broker count. These values make this worked example evenly divisible. Actual per-partition throughput still needs a benchmark.
Verify the fixed-storage layout
| Per-broker check | Estimated load | Target budget |
|---|---|---|
| Disk writes | 9.23 MB/s | 70 MB/s |
| Network inbound | 9.23 MB/s | 700 MB/s |
| Network outbound | 9.23 MB/s | 700 MB/s |
| Retained data | 5.58 TB | 5.6 TB |
The throughput checks divide 3,600 MB/s by 390 brokers. The storage check divides 2,177.28 TB by 390. These are averages under balanced placement.
One cluster across three availability zones
Configure AZ-aware replica placement so every partition has one copy in each AZ. Its leader can be in any AZ, with leaders balanced across the cluster. 390 is the total count across all three AZs. Replication is already included in the sizing.
This verifies steady-state budgets under the assumptions. Failure recovery, historical replay, CPU, data skew, and actual partition performance remain separate checks. A 70% target by itself does not prove survival of an AZ outage.
Evaluate the deployment choices
The result is 390 brokers because this example fixes storage at 8 TB each and retains all seven days on broker storage. Larger storage or tiered storage could reduce the broker count.
Keeping 54 brokers would require about 57.6 TB provisioned per broker at 70% occupancy. That is a generic hardware alternative, not an available MSK Standard configuration.
| AWS option | Storage implication |
|---|---|
| MSK Standard | Up to 16 TiB of broker storage. A 390-broker cluster also exceeds the default cluster quota. |
| MSK Standard with tiered storage | Separate local retention from total retention; size local and remote storage separately. |
| MSK Express | Storage scales automatically; no per-broker disk size to provision. |
| Kafka managed on EC2 | Storage depends on the instance and attached volumes; validate bandwidth and recovery time. |
AWS capacities and quotas can change. Consult the linked service documentation before selecting a deployment. TB in the calculations is decimal; AWS's TiB limit is binary.
Size Amazon MSK Express
Use 1,200 MB/s of producer traffic, one full-stream consumer group per topic, and three copies. Size against AWS's published service throughput for express.m7g.4xlarge.
| Published sustained capacity | Per broker |
|---|---|
| Producer ingress | 124.9 MB/s |
| Consumer egress | 249.8 MB/s |
Producer ingress is the supported rate for accepting producer data, with allowance for storage, replication, and background operations. It is not a raw disk-write or network-interface specification. AWS manages the underlying storage; consumer egress remains a separate check. AWS Express sizing guidance.
Use our 70% planning target on the published sustained rates Ingress: 1,200 ÷ (124.9 × 0.70) ≈ 13.73 → 14 brokers Egress: 1,200 ÷ (249.8 × 0.70) ≈ 6.86 → 7 brokers Take the larger count and round for three AZs 15 brokers total = 5 per AZ
Do not multiply the 1,200 MB/s by replication factor again. The Express service limits already account for the work behind producer ingress.
Verify per-broker client traffic
Producer ingress: 1,200 ÷ 15 = 80 MB/s Planning budget: 124.9 × 70% = 87.43 MB/s Consumer egress: 1,200 ÷ 15 = 80 MB/s Planning budget: 249.8 × 70% = 174.86 MB/s
Both fit under balanced traffic. This is a capacity estimate, not an executed workload benchmark or proof of outage recovery.
Retention affects storage cost, not this broker count
Express storage scales automatically. One-day and seven-day retention both use the same 15-broker throughput estimate. We do not reserve 8 TB disks or divide retained bytes by a per-broker disk size. Express storage documentation.
Revisit partitions independently
Using our assumed 10,000 events/s per consumer, the minimum is 100 partitions for A and 20 for B. An illustrative choice is 135 for A and 27 for B, with additional room for consumer parallelism. More partitions remain valid if the workload needs them.
At 162 logical partitions and RF 3, there are 486 copies, averaging 32.4 per broker across 15 brokers. Exact divisibility is not required. Validate partition traffic, key balance, processing headroom, and service limits.
Price MSK ExpressPrice the Standard baseline
Use US East (N. Virginia), USD on-demand prices, and a 30-day month of 720 hours. Assume 1.2 GB/s continuously, no compression, and a full retention window already accumulated. These estimates exclude client transfer, producer and consumer compute, monitoring, PrivateLink, taxes, and discounts.
Earlier sizing uses decimal GB and TB. The calculations below convert actual bytes to GiB for AWS's billing convention. This changes the unit labels, not the workload.
Monthly ingestion 1.2 × 10⁹ bytes/s × 86,400 × 30 ÷ 2³⁰ ≈ 2,896,786 GiB One-day retained logical data 1.2 × 10⁹ bytes/s × 86,400 ÷ 2³⁰ ≈ 96,560 GiB Seven-day retained logical data 96,559.52 × 7 ≈ 675,917 GiB
MSK Express: about $73,875/month for one day
| Charge | Rate / basis | 1 day retained | 7 days retained |
|---|---|---|---|
| 15 Express brokers | 15 × 720 × $3.264/hour | $35,251 | $35,251 |
| Ingestion | $0.01 per billing GB | $28,968 | $28,968 |
| Retained logical storage | $0.10 per billing GB-month | $9,656 | $67,592 |
| Monthly subtotal | Rounded to nearest dollar | $73,875 | $131,811 |
Express bills retained logical data once. Do not multiply its storage charge by RF 3. Reducing retention changes the storage bill; it does not change the incoming traffic or the broker estimate. MSK does not charge internal broker replication transfer, but client data transfer may incur charges.
Standard baseline: about $347,872/month for seven days
To price the earlier generic fixed-storage example, assume 390 kafka.m7g.large brokers with 7,451 GiB each, approximately 8 decimal TB. The instance choice needs workload validation and this fleet requires quota approval.
| Charge | Calculation | Monthly cost |
|---|---|---|
| Brokers | 390 × 720 × $0.204/hour | $57,283 |
| Provisioned storage | 390 × 7,451 GiB × $0.10/month | $290,589 |
| Subtotal | Seven-day fixed-storage baseline | $347,872 |
This Standard estimate includes provisioned space for all replicas and unused capacity. It is not an optimized Standard deployment. Standard with tiered storage needs its own sizing and cost comparison before choosing a service.
Rates were checked against AWS's US East price list, published September 11, 2026. AWS labels these usage rates GB and GB-month; the byte conversion here follows its GiB-based sizing convention. Actual bills include metadata and other measured usage. An initially empty cluster has lower storage usage while retention fills.
AWS MSK pricing · AWS regional price list · AWS billing and sizing worked examples
Add brokers to a running cluster
Adding the broker is the easy part. Kafka does not rebalance on its own, so the work is moving data onto it.
1. Start the broker
Give it a unique node.id and point it at the same cluster (in KRaft mode, the same controller quorum). It registers and appears in the cluster metadata, holding nothing. Existing partitions stay where they are.
2. Only new data lands on it for free
New topics, and new partitions of existing topics, can be placed on the new broker. Old partitions never move by themselves: a cluster you grow still carries all its load on the original brokers.
3. Move partitions with a reassignment
kafka-reassign-partitions.sh --generate # propose a plan for the chosen topics kafka-reassign-partitions.sh --execute --reassignment-json-file plan.json --throttle 50000000 kafka-reassign-partitions.sh --verify # confirm it finished, then remove the throttle
The new broker joins as an extra follower and copies the partition's whole log from the leader. Once it has caught up and is in the in-sync replica set, it replaces the old replica, which is dropped. Producers and consumers keep working throughout.
4. Throttle the copy
Moving a replica copies all of its retained data over the network and onto the new disk. Without a throttle, that traffic competes with producers and consumers and latency rises.
5. Rebalance leaders
After replicas move, run a preferred leader election (or leave auto.leader.rebalance.enable on) so leadership, and with it client traffic, spreads onto the new brokers too.
Worked example: 390 to 420 brokers
Add 30 brokers, 10 per AZ, so placement stays even across the three AZs.
Data per broker after the move 2,177.28 TB ÷ 420 ≈ 5.18 TB Data copied onto the 30 new brokers 30 × 5.18 TB ≈ 155.5 TB, about 0.4 TB sent by each existing broker Throttle: the new broker's disk-write budget from Step 02 100 MB/s × 0.70 = 70 MB/s, so a 50 MB/s throttle leaves room for live writes Time to fill one new broker 5.18 TB ÷ 50 MB/s ≈ 103,700 s ≈ 29 hours
Growing this cluster takes more than a day of copying, run in parallel across the 30 new brokers.
The 2,340 stored copies do not divide evenly by 420: 240 brokers hold 6 copies and 180 hold 5. Partitions are the unit of movement, so balance is only as fine as the partition count allows.
With MSK Express
Express keeps broker storage managed and separate from the broker count, so a reassignment has far less local log to copy than the fixed-storage example. The broker count still comes from throughput: adding one broker per AZ takes this example from 15 to 18 and lowers per-broker client traffic from 80 MB/s to about 67 MB/s. Check the MSK documentation for how Express rebalances partitions onto new brokers.
In practice
| Option | How rebalancing works |
|---|---|
| Cruise Control | LinkedIn's open-source tool computes and runs the reassignment plan from disk, network and CPU load. |
| MSK Standard | Raise the broker count in the console, in multiples of the AZ count, then rebalance with Cruise Control or the reassignment tool. |
| MSK Express | Storage is not tied to the broker, so there is far less data to move. |
| Tiered storage | Only the local, hot part of each log moves, which cuts reassignment time sharply. |
What interviewers listen for. Partitions are the unit of movement: you can move one but never split one, so choose enough partitions up front for the brokers you will eventually run. Adding partitions later is a different operation, and keys then hash to new partitions, which breaks per-key ordering across the change. Scaling out is not instant: new brokers help only after data is copied onto them, so add capacity ahead of a peak, not during one.
Explain the deployment choice
"The topics produce 1.2 GB/s combined. Using AWS's published throughput for express.m7g.4xlarge, I estimate 15 brokers with headroom across three AZs. Storage scales separately. At sustained traffic, one-day retention costs about $74,000 per month, versus $132,000 for seven days, before client transfer and application costs. I'd validate throughput and size consumers and partitions for the processing workload."
"The topics produce 1.2 GB/s combined. Under our assumed disk and network capacities, throughput needs about 54 brokers with headroom. Seven-day retention raises the requirement to 390 brokers with 8 TB each. I'd choose partitions for processing and storage placement, then evaluate larger disks or tiered storage."
Workload → deployment model → throughput capacity → retention and billing → partitions and consumers → placement and failure checks.
Sources and scope
The fixed-storage hardware and consumer processing rates are illustrative assumptions. Express throughput and AWS prices use the linked published figures. No workload benchmark was performed.