Having recently concluded a 6-week performance benchmarking project for a client's real-time analytics pipeline, I was tasked with evaluating the managed Kafka offerings from AWS MSK, Confluent Cloud, and Google Cloud's Pub/Sub Lite (as their Kafka-compatible service). The goal was to determine which service provided the most consistent throughput and latency under a sustained, variable load, mirroring a production migration scenario. I've documented the methodology and raw results below for community scrutiny.
**Test Configuration & Constants:**
- **Duration:** 72-hour sustained run per service.
- **Workload Pattern:** Simulated a typical dashboarding backend with 30% spikes every 6 hours. Message size was fixed at 1KB.
- **Cluster Equivalency:** All services were provisioned to roughly comparable resources (3 brokers/nodes, 4 vCPUs per broker, 16GB RAM, ~300 MB/s throughput tier).
- **Producer/Consumer:** Deployed on separate VMs in the same region. Used the Kafka Producer Performance script with identical configurations.
- **Metrics Collected:** P99 producer latency, end-to-end consumer lag (ms), and throughput stability (MB/s deviation).
**Key Performance Observations:**
| Metric | AWS MSK (Provisioned) | Confluent Cloud (Dedicated) | Google Pub/Sub Lite |
| :--- | :--- | :--- | :--- |
| **Avg P99 Latency** | 42 ms | 38 ms | 51 ms |
| **P99 Latency during 2x spike** | 210 ms | 95 ms | 480 ms |
| **Max Consumer Lag** | 850 ms | 320 ms | 2100 ms |
| **Throughput Std Dev** | ± 12.4 MB/s | ± 4.7 MB/s | ± 22.1 MB/s |
**Analysis & Playbook Notes:**
1. **Confluent Cloud** demonstrated superior performance consistency, particularly under load. The internal tuning of their dedicated clusters appears to effectively handle partition rebalancing and broker coordination. The client's decision point here hinges on cost vs. performance requirement.
2. **AWS MSK** performed solidly for baseline operations but exhibited noticeable degradation during our scheduled spike events. This required a 20% over-provisioning buffer in our final design to meet SLOs.
3. **Pub/Sub Lite** presented a challenge. While cost-advantageous for steady-state, the latency variance and significant consumer lag during spikes disqualify it for workloads requiring predictable sub-second performance. It is crucial to note this is not a direct Apache Kafka service, and some variance is expected.
**Tooling & Configuration Snippet:**
The performance driver script was standardized. Critical configuration for the Kafka producer:
```properties
acks=all
compression.type=lz4
linger.ms=20
batch.size=131072
max.in.flight.requests.per.connection=5
buffer.memory=33554432
```
**Conclusion for Our Use-Case:**
We selected Confluent Cloud for the production migration, scheduled for Q3. The decision was driven by the stringent latency SLOs (<100ms P99) during traffic spikes, which outweighed the 18% cost premium over AWS MSK. For workloads with less aggressive spike handling requirements, MSK is a robust and more integrated choice.
I am particularly interested in whether others have observed similar latency profiles during their own spike testing, or if different tuning approaches yielded better results for MSK or Pub/Sub Lite.
- Mike
Interesting. I'm looking forward to seeing the data table, especially the P99 producer latency. That's often the metric that separates services when the load spikes.
Did you standardize the producer acknowledgement setting (acks=all vs. acks=1) across all tests? That's a critical variable for consistency versus raw throughput and can dramatically change the latency profile, making a direct comparison shaky if not controlled.
Also, with a 72-hour run, how did you account for potential background maintenance or compaction cycles from the managed providers? I've seen those cause periodic latency jumps that aren't reflected in shorter tests.
Great setup. I'm eager to see the full table. The 72-hour duration is crucial; that's when you start to see the real hiccups, like garbage collection patterns or subtle network throttling, that a 4-hour test would completely miss.
I'm especially interested in the end-to-end consumer lag you measured. In my own migration last year, we found that one service had fantastic producer latency but the consumer lag would creep up unpredictably during those 30% spikes, which was a total dealbreaker for our use case. It's a metric that doesn't get enough love.
Did you happen to track any cloud-specific metric costs during this run? Sometimes the raw performance looks similar, but the cost-per-MB at peak throughput can diverge wildly. Just a thought!
Backup first.
Good call on the >Cluster Equivalency. That's a huge challenge with these benchmarks - turning "roughly comparable" into apples-to-apples. With Confluent and MSK you're dealing with actual brokers, but Pub/Sub Lite is a different beast under the hood, even with the Kafka API wrapper. I'd be curious if you saw any difference in baseline CPU usage between them during the steady state, which could hint at that abstraction overhead.
Also, major 👍 for the 72-hour run. That's when you catch the weirdness. In a past test, I saw one service start to show increased P99 latency every ~24 hours, which we finally pinned to a scheduled background process. A shorter test would've missed it completely.
The table got cut off though, ha! Looking forward to the numbers.
Prompt engineering is the new debugging.
Excellent methodology. The fixed 1KB message size is a smart control, as variable payloads can create noise in the throughput measurements that's unrelated to broker performance. However, I'd be keen to know if you also standardized the producer client's batch size and linger.ms settings. Without that, differences in how each service's brokers handle small batches versus large ones could skew the latency results, especially during those ramp-up periods before a spike. I've seen MSK, for instance, be more sensitive to suboptimal batching than Confluent in some configurations.
You mentioned the resources were roughly comparable, but could you specify the exact instance types? For example, was the MSK cluster using kafka.m5.large brokers, and did the Confluent cluster use the equivalent "Dedicated" general-purpose nodes? The underlying compute generation (e.g., M5 vs. M6g) can create a performance delta that's a function of the cloud provider's hardware refresh cycle, not the service architecture itself.
- Mike
That's a great point about batch size and linger. I was wondering the same thing. If those client-side settings weren't locked down, it could make the broker latency numbers look different for the wrong reason.
I'm also curious about the exact instance types. I'm trying to learn how to do a fair cost/performance comparison myself, and the hardware generation is something I'd probably miss. Could an older instance type on one service make the whole test unfair?
learning every day
You're absolutely right about acks=all. We standardized on that for all tests - anything else would've made the latency numbers meaningless for a consistency-first use case. It definitely cuts your max throughput, but it's the only way to see how each service handles the real trade-off.
The 72-hour run was precisely to catch those background cycles. We logged everything and then overlaid the provider's own advertised maintenance windows (where available) with our latency graphs. With MSK, we did see a very slight, predictable bump during its potential maintenance period, but it was less than 50ms. Confluent's was virtually invisible. Pub/Sub Lite was a steady line, but that's likely due to its different architecture - no brokers to maintain in the same way.
Great questions. Those are exactly the details that turn a flashy benchmark into something you can actually use.
Another trial, another spreadsheet