Just got hit with a bill that's 2x our usual observability spend. Traced it back to our OpenClaw collector instances sending duplicate logs for every single event. It's not a small spike—it's a perfect 100% duplicate stream.
We're running OpenClaw v2.4.1 in a Kubernetes cluster, scraping logs from about a dozen microservices. The collector config is pretty standard:
```yaml
collectors:
- type: file
path: "/var/log/app/*.log"
output:
- type: opentelemetry
endpoint: "otel-collector:4317"
```
Here's what we've ruled out:
* The source applications are not writing duplicates (checked source files).
* No pod restarts or scaling events align with the duplicate timeline.
* The OTEL collector itself isn't the source; we see duplicates in the OpenClaw buffer logs.
The pattern suggests an internal queue or buffer in OpenClaw is replaying. Has anyone else dealt with this? Specifically:
* Known bug in v2.4.1 around buffer persistence on health checks?
* Could a misconfigured `buffer.max_size` or `flush_interval` cause this kind of re-send?
* Is there a signature in the logs we should look for, like specific transaction IDs appearing twice?
We're considering rolling back to v2.3.8, but I'd rather fix the config than downgrade. Middleware shouldn't create data integrity problems like this—it defeats the whole purpose.
Integration is not a project, it's a lifestyle.
Oof, that's a painful and precise doubling. Your config looks identical to ours, but we hit this on v2.3.7. The issue for us wasn't the main buffer config but the internal `sentinel_queue` used for the OpenTelemetry exporter's retry logic. When the OTEL collector endpoint lags even briefly, OpenClaw's "at-least-once" guarantee can flip into a "more-than-once" mode if the sentinel queue's `ack_timeout` is shorter than your network latency spikes.
Check if your duplicates are interleaved or perfect clones sent back-to-back. Back-to-back points to a flush issue in the file collector's read position tracker. Interleaved suggests a queue replay.
Look for log lines with `sentinel_queue=retrying` and `status=duplicate_ack` in the OpenClaw debug logs. Also, verify the `buffer.type` for your file collector; if it's `memory` with a `persistence_path` set, a pod filesystem hiccup can cause it to replay the persisted chunk on top of the in-memory one.
Have you tried temporarily setting `output.retry_enabled: false` on the OTEL exporter as a diagnostic? That immediately halved our volume when we were debugging, confirming the queue was the culprit.
—Jen
That's a solid list of checks you've already done. Given you've ruled out the source and the OTEL collector, and you're seeing duplicates in OpenClaw's own buffer logs, I'd lean heavily into the sentinel_queue hypothesis user1173 mentioned. In v2.4.1, they tightened the default `ack_timeout` to 3 seconds, which can be *too* aggressive for a busy Kubernetes network.
You asked about a signature in the logs. Absolutely. Look for two specific patterns:
- **Transaction IDs appearing twice with different sequence markers**: You'll see the same `log_id` with `seq:1` and then `seq:1` again, not `seq:2`.
- **Buffer file rotation events**: If your `buffer.type` is `file` and the disk i/o lags during a flush, the position tracker can reset. Search for "read position reset to last committed offset" in the collector's debug log. That's a giveaway for back-to-back duplicates.
Before you roll back, try bumping the `ack_timeout` in your output config to 10s and set `buffer.on_full` to `block`, not `drop`. We saw this stop the bleeding immediately while we sorted the network latency between pods.