Hi everyone. I've been running into a persistent and somewhat tricky issue with the Arize AI Java SDK in our Kubernetes environment, and I'm curious if others have faced similar hurdles or have any insights.
We're running a suite of Java (Spring Boot) microservices in a K8s cluster, instrumented with the Arize SDK for model observability. The problem manifests in two ways:
1. **Memory growth:** Certain pods exhibit a steady, slow memory increase over days, correlating with the number of predictions logged. It feels like a slow leak, not a rapid OOM.
2. **Thread pool exhaustion:** We've seen intermittent `RejectedExecutionException` logs coming from the SDK's internal async dispatcher, especially during rolling updates or scale-up events.
Our configuration is fairly standard. We're using the async client with a setup similar to this:
```yaml
# application.yaml snippet
arize:
apiKey: ${ARIZE_API_KEY}
spaceKey: ${ARIZE_SPACE_KEY}
# We've tried both true and false here
asyncMode: true
# Increased from defaults
maxConcurrentRequests: 32
queueSize: 10000
```
And in the pod spec, we have reasonable memory limits and requests set.
My hypothesis is that the async queue might be filling faster than it can drain under specific load patterns, or perhaps there's a subtle issue with OkHttp client or dispatcher lifecycle not playing nicely with the Spring Context shutdown in a container environment. We've tried adjusting `queueSize` and `maxConcurrentRequests`, but the improvements were temporary.
Has anyone else battled this combo (Java SDK + K8s)? Specifically:
* Did you have to implement a custom `Sender` or `Client` configuration for better control?
* Are there known patterns for gracefully shutting down the Arize client during a pod termination event to prevent queued logs from being lost or causing delays?
* Any JVM tuning (GC settings, etc.) that proved particularly helpful?
I'm happy to share more details about our deployment patterns or error logs. This feels like a complex trade-off between throughput, reliability, and resource efficiency in a dynamic orchestrated environment.
—Chris
Prod is the only environment that matters.