Hey folks! I'm deep in the middle of evaluating a few observability platforms for a potential switch, and the biggest question mark for me right now is **log ingestion throughput**. The sales decks all promise the moon on scalability, but I've been burned before by assuming those numbers would hold up under my actual, messy, real-world log patterns.
So, I'm trying to design a rigorous test for the proof-of-concept phase that goes beyond just "send a bunch of logs." I want to simulate the chaos of production without actually breaking anything. How are you all stress-testing this?
Here’s my current thinking for a test plan, but I'd *love* to hear how you've approached this or where I might be missing something:
* **Baseline with Realistic Log Shapes:** Instead of blasting identical log lines, I'm scripting a log generator that mimics our actual variety:
* Mix of structured (JSON) and plain-text application logs.
* Variable log line lengths (from terse health checks to massive stack traces).
* Fluctuating cardinality in key attributes (like `user_id`, `request_id`, `transaction_id`). I'll gradually ramp this up to see where things start to groan.
* Simulating "bursty" traffic patterns—quiet periods followed by sudden spikes, because that's when the bills (or the dropped logs) really hurt.
* **Measuring Beyond Just "Logs/Sec":** I plan to track a few key metrics during the ingestion blast:
* **End-to-end latency:** From log generation to being queryable in the platform's UI. This is the real user-experience metric.
* **Agent/Collector Resource Footprint:** CPU and memory usage on our side during sustained high throughput. A hungry agent can be a hidden ops cost.
* **Error & Duplication Rates:** Checking for any ingestion errors, backpressure, or mysterious log duplication under load.
* **Query Performance Under Load:** Running simple and complex queries on the freshly ingested data *while* the ingestion is still running. Does query latency degrade?
* **The "Gotcha" Tests:** These are the fun, experimental ones:
* Suddenly killing the ingestion agent or a network link to see how the platform handles disruption and backlog.
* Switching from a high-volume, low-cardinality stream to a lower-volume, ultra-high-cardinality stream mid-test.
* Validating the actual ingested data against what was sent—a simple spot-check for fidelity.
My main hangups are around tooling. I'm leaning towards a custom script using something like `fluent-bit` or `vector` as the generator, but I'm curious if anyone has used a dedicated load-testing tool for logs they'd recommend.
What has your experience been? What thresholds or behaviors made you rule a platform in or out during a POC? Did you discover any clever ways to simulate truly pathological logging behavior?
Try everything, keep what works.
I'm a project manager at a 150-person SaaS company, and we run our own microservices for our core platform, so we're currently pushing around 50 GB of logs daily to our existing provider.
Here's what I'd focus on testing based on what we learned the hard way:
1. **Test with Real Data Samples, Not Just Volume.** We made the mistake of just blasting random log lines. You should pull a few hours of actual production logs (scrubbed of PII) and replay them during the POC. The structure, fields, and randomness of your real data matter more than a vendor's synthetic test. In our case, one platform handled our JSON cleanly but choked on multiline Java exceptions another sent.
2. **Measure the End-to-End Latency, Not Just Ingestion.** Ask the vendor for their "data freshness" metric from ingest to query readiness, and verify it yourself. We saw one platform accept logs at 20k EPS but take over 90 seconds to make them searchable, which killed our incident response.
3. **Simulate a Traffic Spike and Sustained Load.** Gradually ramp up to your peak expected rate (like 2x your normal volume), hold it for 30-60 minutes, and then watch what happens when you drop back to normal. Does the backlog clear quickly? Does query performance degrade during ingestion? One vendor we tested had serious query slowdowns when ingestion was over 70% of their stated limit.
4. **Check the Actual Cost Calculation.** Their throughput limit might be fine, but their pricing model might not. Be specific: is the limit in GB/day, events/second, or both? Do they count the full JSON payload toward your volume, or just the parsed message? For one vendor, a 1KB log line with high-cardinality labels counted as nearly 2KB billed, which blew our estimate.
I'd recommend setting up a test with a vendor who gives you a dedicated ingest endpoint for the POC, not a shared one. My specific choice would depend on your budget and team size. Tell us your expected daily log volume and whether you need compliance certifications like SOC2, and I can point you to which of our finalists worked for those needs.
Excellent point about **testing with real data samples**. I've found the variance in log structure can expose parsing inefficiencies that synthetic tests miss completely. One caveat: when replaying scrubbed logs, ensure your scrubbing process doesn't inadvertently normalize the data, like turning all string lengths to a fixed size. You need to preserve the original byte distribution and field entropy to get a true throughput test.
Your second point on **end-to-end latency** is critical and often obscured by vendor metrics. I'd add that you should also test query performance *during* the sustained load you mentioned. Some systems ingest quickly but create indexing bottlenecks that only manifest under pressure, degrading query times precisely when you need them most. The 90-second delay you saw is a perfect example of a system that's optimized for the ingest buffer, not the full pipeline.
Garbage in, garbage out.