So, we finally migrated our core summarization service off our old BERT-based setup to OpenClaw's shiny new flagship model, hosted on their Azure offering. The sales pitch was compelling: sub-100ms p95 latency for our average payload, with "near-linear scaling" on their provisioned throughput. Our internal SLA target is 150ms. Seemed like a solid upgrade path.
The reality, after two weeks of load testing and gradual ramp-up, has been... different. Our p95 latency settled around 220ms, with periodic spikes to 400ms+ that correlate with nothing in our own logs. Their dashboard shows all green checkmarks, of course. Support's initial response was the classic "your payloads might be larger than expected." We provided the logs. Then it was "cold starts on scaled instances." We showed them the sustained traffic pattern that should prevent that.
The real kicker? We managed to trace one of the spikes. Their Python SDK has a default timeout configuration that *doesn't* match their gateway's actual behavior. We were getting silent retries. Here's the snippet we had to implement to even see the issue:
```python
from openclaw import AzureClient
import httpx
# Their default client setup hides the retry logic
client = AzureClient(
endpoint=endpoint,
credential=credential,
# Without these overrides, you're blind
timeout=httpx.Timeout(5.0, connect=10.0),
transport=httpx.HTTPTransport(retries=1) # They were doing 3!
)
```
After we pinned the retries and set explicit timeouts, the error logs finally showed the 503s that were being masked. The latency "improvement" we then got was just us failing faster, not them processing quicker.
We're sticking with it for now because the accuracy *is* a step-change better for our use case. But we're having to build in a much larger latency buffer and implement our own queuing layer in front of it, which negates a lot of the "managed service" benefit. Would we renew at this price point? Only if the next quarter shows measurable improvement on their infrastructure stability. The accuracy saves us engineering time on post-processing, but the latency unpredictability is costing us user satisfaction.
~d