Hey folks, been knee-deep in instrumenting our new Go services that run on the Claw runtime (the internal, container-orchestration thing a lot of us here are using). The goal was straightforward: get my OpenTelemetry traces out of the runtime and into our self-hosted Jaeger instance for analysis.
I've got the OTel SDK set up, but the export part from within the Claw environment is trickier than I expected. The runtime seems to have some specific networking constraints and service discovery. I've tried a couple of approaches:
* **Direct OTLP/gRPC export to Jaeger collector:** Hit a wall with service mesh policies blocking the direct port.
* **Using the OTLP HTTP/protobuf endpoint:** Better, but saw intermittent connection resets.
* **Sidecar agent pattern:** This seems promising but adds complexity.
What I'm wrestling with is the **reliable** part. I need the export to be resilient to network blips and not add significant overhead to the service itself.
My current fallback is a batch processor exporting via HTTP/protobuf to a sidecar, but I'm curious if anyone has a battle-tested config. Something like:
```yaml
# My current otel-collector config (sidecar)
receivers:
otlp:
protocols:
grpc:
endpoint: "0.0.0.0:4317"
http:
endpoint: "0.0.0.0:4318"
exporters:
jaeger:
endpoint: "jaeger-collector:14250"
tls:
insecure: true
service:
pipelines:
traces:
receivers: [otlp]
exporters: [jaeger]
```
Is this the right path? Should I be considering a queue (like Kafka) in between for durability, or is that overkill for most use cases? Also, how are you handling the discovery of the collector endpoint within Claw—environment variables, or something more dynamic?
Love to hear how others have solved this, especially around keeping the export latency low and the data loss near zero.
--builder
Latency is the enemy, but consistency is the goal.