I'm evaluating Cribl Stream as a passthrough processor to normalize logs before our SIEM. The goal was simple: reduce some ingestion costs by filtering noise and dropping high-cardinality debug events we never query.
My architecture is straightforward: existing agents (FluentD, Winlogbeat) now point to a Cribl worker group instead of directly to the SIEM (Splunk Cloud). The worker group applies basic filters, then routes the processed data onward.
The problem: I'm measuring a consistent 80-100ms increase in end-to-end latency compared to the old direct agent-to-SIEM path. This is enough to cause alerting delays that my security ops team is complaining about.
My route configuration is barebones:
* Source: a dedicated TCP port for each agent type.
* Pipeline: A single filter expression to drop events matching `DEBUG` and a few known chatty sources.
* Destination: Straight to the Splunk HEC endpoint, same as before.
I'm not doing heavy parsing, enrichment, or lookups. Just a simple filter.
My questions:
* Is this latency overhead typical for a single filter operation, or does it point to a misconfiguration?
* Could the worker group's autoscaling (in AWS) be introducing connection overhead? I've pinned the instance count to rule that out.
* Should I be looking at batching or buffer settings on the source or destination to mitigate this, even if it means slightly higher memory use?
I need to justify the TCO, and if the trade-off is paying for Cribl plus adding 100ms to every critical log, the math doesn't work. The filtering savings are real, but the latency is a hard constraint.
tldr: Simple Cribl filter pipeline adds 100ms latency. Is this expected, or am I missing a configuration knob?
Your CFO thanks me.
80-100ms added by a simple filter sounds off. Typical overhead for a passthrough setup like yours should be in the low single-digit milliseconds, maybe 10ms if the worker is under-provisioned.
> Could the worker group's autoscaling be introducing connection
That's where I'd look first. If your workers are scaling in/out in AWS, you're likely seeing cold starts or connection churn. Check if the latency spikes correlate with scaling events in your metrics. Also, make sure your worker group's instance type isn't CPU-starved; a t3.small trying to handle TCP streams and filter eval can get chatty.
Have you checked the destination queue metrics in the Cribl UI? If events are backing up there even briefly, that's your smoking gun. Sometimes the default batching for the Splunk HEC destination introduces more delay than you'd expect, especially if you didn't tune the batch size and timeout from the agent->SIEM days.
That's a significant jump for a simple filter, and I don't think the route config is the culprit. I agree with the later post that autoscaling cold starts could be a factor, but I'd check two other places first.
One, verify the network path. Are your agents, Cribl workers, and Splunk HEC endpoint all in the same region/AZ? Adding a middle hop can sometimes route traffic through a different AZ, adding that kind of latency. A quick traceroute from a worker pod to the Splunk endpoint can be revealing.
Two, look at your destination batching. The Splunk HEC destination has a `Flush Period (sec)` setting. If it's buffering events for too long to create larger batches, you're adding queue time. Try setting it to 0.1 seconds temporarily as a test to see if latency drops - you can tune it back up later for throughput.
Good point about the network path. The traceroute idea is smart, but what if the extra latency isn't constant? Could an internal load balancer between the agents and the worker group be adding inconsistent hops? That might explain the spikes.
Great call on the network path check. I've seen that bite us when we moved our workers to a different VPC for compliance reasons - we assumed everything was peered efficiently, but the actual traffic path added three extra hops and about 70ms. A quick `mtr` from a worker is my go-to now, not just a ping.
> look at your destination batching
Absolutely second this. The default flush period is often too high for alert-sensitive data. We run ours at 0.05 seconds for security logs, but had to bump up the batch size to compensate. Just watch your worker group's memory if you have a high EPS, because those smaller, frequent batches can pile up under heavy load.
Data is the new oil
Yeah, that's way too high for a simple filter. I've got a similar setup for email campaign logs and it adds maybe 2-3ms.
Did you check the worker group's TCP keep-alive settings? If your agents are opening new connections for each small batch, that handshake overhead could be adding up fast, especially if there's any network distance.
Also, what's your filter expression look like? If you're using a regex on the raw event, that can be slower than using a structured field that's already parsed.
That's a classic case of adding a hop and expecting zero cost, which never works out. I'll bet good money the issue isn't your filter expression; it's the extra TCP session.
You're now forcing every event flow through two separate TCP connections: agent->Cribl, then Cribl->Splunk HEC. Each one has its own buffering, Nagle's algorithm, and ACK delays. That alone can add tens of ms, especially if your batches are small.
> Could the worker group's autoscaling (in AWS) be introducing connection...
Absolutely it could. A cold start on a new worker means fresh TCP sessions, which is pure latency. But even without scaling, you've doubled the network stack overhead. The filter is probably costing you 2ms. The rest is connection plumbing.
been there, migrated that