I've seen this pattern a few times during on-call rotations: an internal service or agent works fine for the first few requests, then response times jump and plateau at roughly double the baseline. It's almost always a resource exhaustion issue that only surfaces under sustained load, which is why it slips past simple health checks.
The most common culprit I've found is a connection pool or thread pool that's too small. The first few requests grab the initial, fresh resources. Once those are saturated, incoming requests wait in a queue, adding latency. By the time you hit request #10, the queueing delay is roughly equal to the processing time, hence the "double" effect.
Let's break down where to look. First, check if your agent or its dependencies have configurable pools. For a Java-based agent, you might see something like this in the config:
```yaml
# application.yml snippet
http:
client:
max-connections: 20
max-connections-per-route: 10
connect-timeout: 5s
socket-timeout: 30s
```
If `max-connections-per-route` is set to, say, 5, and your processing isn't instantaneous, the 6th-10th requests will wait. The same logic applies to database connection pools (e.g., HikariCP `maximumPoolSize`), gRPC channels, or any bounded worker thread pool.
To confirm, the dashboard panel I live by for this is a simple `rate()` of request duration, segmented by quantile, over a short window. Pair it with a graph of active threads or connections.
```promql
# Prometheus query example
histogram_quantile(0.95, rate(agent_request_duration_seconds_bucket[5m]))
histogram_quantile(0.50, rate(agent_request_duration_seconds_bucket[5m]))
```
Plot those two quantiles together. If the 95th starts diverging sharply from the median after a load ramp, you're likely looking at queueing. The next step is to correlate it with a metric like `thread_pool_active_threads` or `http_client_active_connections` hitting its max.
What's the underlying tech stack for your agent? If you can share the relevant pool configurations and a snippet of the latency graph, we can probably pinpoint the tuning knob.
zzz
Sleep is for the weak