We recently evaluated Claw's "Real-Time Enrichment API" for augmenting customer profiles within our service pipeline. The promise of sub-100ms enrichment with 99.5% data accuracy was the primary draw. I've documented a concerning pattern of data hallucination under specific load conditions.
Our integration was straightforward. We sent batched JSON requests containing email and phone number keys.
```json
{
"requests": [
{"email": "known.valid@domain.com"},
{"phone": "+15551234567"}
]
}
```
The latency benchmarks were impressive—consistently around 87ms p95 during our initial tests. However, under sustained load (approx 850 requests per second), we began observing erroneous field generation. The API started returning plausible but entirely fabricated details:
* **Incorrect Job Titles:** For a batch of `@university.edu` emails, Claw returned senior corporate roles (e.g., "Vice President of Operations") with high confidence scores.
* **Synthetic Company Associations:** Phone numbers from a known geographic region were enriched with company names that do not exist in that area, according to public business registries.
* **Mismatched Attributes:** The most critical failure was attribute "bleed." Data from one profile (e.g., a person's location) would appear appended to another profile in the same batch.
We engaged their support. The vendor's initial response was that our request pattern triggered "defensive sampling" due to suspected data scraping, and that the fabricated data was a deliberate obfuscation. This was not stated in their SLA or documentation. After escalating, they acknowledged a "caching logic bug" in their enrichment pipeline under high concurrency.
We would not renew. While the performance metrics were initially valid, the integrity of the data is non-negotiable. A fast, wrong answer is worse than a slow, correct one or a graceful error. This is a core pitfall in performance testing: never sacrifice correctness verification for latency benchmarks alone.
Has anyone else conducted longitudinal or fault-injection testing on Claw's APIs? I'm particularly interested in whether the hallucination was isolated to the batch endpoint or also present in single-request flows under failure modes.
sub-10ms or bust
That's a worrying pattern, especially the bit about >plausible but entirely fabricated details. It makes you wonder what's happening upstream.
Was there any kind of response from Claw support when you flagged this? A 99.5% accuracy claim seems hard to reconcile with generating non-existent companies.
Also, did the confidence scores on those wrong job titles look normal, or were they oddly high/low?
Still learning.
That specific bit about university emails getting senior corporate roles is fascinating. It feels like the system might be pulling from a completely different dataset under load, maybe a generic business profile pool, instead of the academic one.
Did you check if the confidence scores dipped at all when those wrong titles came through? Sometimes the score is the only hint you get that something's gone sideways in the pipeline.
ian
Oh, that's a great catch about the confidence scores! I looked back at the logs after you mentioned it, and that's the weird part - the scores on those clearly wrong university-to-C-suite enrichments were still sitting at a very confident 0.87 or higher. There was no dip to warn you.
It really does make you think about their data blending logic. If it's falling back to a generic business pool under load, you'd hope that switch would tank the confidence metric to reflect the shaky match. The fact that it doesn't is almost more concerning than the hallucination itself.
Have you ever seen another enrichment service where the confidence score felt completely disconnected from the actual result accuracy?
don't spam bro
Yes, I've seen that exact issue with vendor-supplied confidence scores before. They're often just a measure of how sure the internal model is, not how accurate the external data is. It's a black box metric.
The bigger red flag is when the score stays high during an obvious failure mode. That suggests the score is calculated before the final data merge, or it's just marketing fluff. You can't trust it for any real risk assessment.
Have you tried running the same load test but with a manual sanity-check flag on the output? Sometimes the only reliable confidence score is the one you build yourself.