Hey everyone, I've been working on a Cribl Stream deployment to route logs from a few Kubernetes clusters to an S3-based data lake. Overall it's been great, but I've hit a frustrating issue: I'm seeing a consistent 2-3% event loss between my source and destination, and there are no obvious errors in the Worker Process logs.
My pipeline is fairly standard:
* Source: A TCP JSON listener.
* Processing: A few Eval functions to add metadata, a Parser for some nested fields, and a Filter to drop specific debug events.
* Destination: An AWS S3 destination with gzip compression.
I've checked the basics:
* Worker Process metrics show no `events_dropped` or `routes_dropped`.
* The source's `events_in` matches my app's sent count, but `events_out` at the S3 destination is lower.
* My Filter function is simple, just dropping events where `level == "debug"`. I've double-checked the condition.
Has anyone else dealt with silent drops? I'm thinking the next step is to add a Passthru destination temporarily to see if events make it through the pipeline, but I'd love to hear your debugging playbooks.
My main questions are:
1. Where are the less obvious places to look for loss? Could it be in the Destination's batching or compression?
2. Are there specific pipeline functions known to cause silent issues with certain data shapes?
3. What's the most effective way to add internal logging/tap points in a pipeline without disrupting flow?
Here's a snippet of my Filter function's condition, in case I'm missing something:
```javascript
// Filter to drop verbose debug logs
if (__event.level === 'debug') {
return null;
}
// All other events flow through
return __event;
```
Any pointers would be much appreciated. I'll share my findings once I get to the bottom of it.
-- Amy
Cloud cost nerd. No, I don't use Reserved Instances.
That silent drop is such a pain. The passthru destination is a great next step to isolate the problem.
One thing that tripped me up before - check the S3 destination's "Max file size" and "Flush period" settings. If events are trickling in slowly, they might be getting aged out before the buffer hits the size limit, causing some to be silently discarded at shutdown or rotation. Maybe try lowering the max file size as a test?
Also, have you looked at the Pack Queues tab under Monitoring? Sometimes events get stuck there if there's a tiny mismatch in the event schema that the destination doesn't like.
Beta tester at heart
Yeah, the buffer aging out before hitting the size limit is a sneaky one. I've seen similar behavior where the flush period was set too high for a low-volume stream. Lowering the max file size did force more frequent writes in my case.
You mentioned the Pack Queues tab - is that where you'd see events that are stuck because the destination rejects them, even if there's no clear error logged?
Containers are magic, but I want to know how the magic works.