I’ve been evaluating You.com’s AI search for a few weeks, primarily as a potential tool for accelerating initial research during post-incident reviews. The promise of cited sources is compelling for an SRE—traceability back to an original document or RFC is everything. Unfortunately, the implementation feels like a distributed system with beautifully instrumented spans that point to entirely wrong services.
The core issue: the citations provided often do not, in fact, contain the information used to generate the answer. It’s not just a minor mismatch; it’s a fundamental breakdown in attribution. This is worse than a vanity dashboard showing p99 latency at 50ms while your phone is ringing off the hook with user complaints.
A concrete example from my testing yesterday:
* **My Query:** "What is the default scrape interval for Prometheus?"
* **You.com's Answer:** "The default scrape interval in Prometheus is 15 seconds." It then cited three sources.
* **Reality Check:** The first cited source was the official Prometheus documentation page on configuration. A `Ctrl+F` for "default" and "interval" on that page reveals no such statement. The actual default is pulled from the `scrape_configs`, and a global default isn't explicitly set in the binary; the common 15s is a convention, not a hardcoded default. The other two sources were tangential blog posts that also didn't state a default.
This isn't a one-off. I’ve seen it with Kubernetes manifest examples and OpenTelemetry collector configuration. The pattern suggests the model is generating a correct-*adjacent* answer, then performing a retrieval step that fetches somewhat relevant documents, and slapping numbers next to sentences with weak correlation. There’s no integrity check between the claim and the source text.
For anyone using this in a professional capacity, this is a critical flaw. It forces you to:
1. Trust the answer at your own peril.
2. Independently verify every cited source, which defeats the efficiency purpose.
3. Assume any unattributed statement is pure hallucination.
Has anyone else subjected this feature to rigorous validation? I’m curious if my experience is the norm and, more importantly, if there’s a discernible pattern to the citation failures. Without fixable accuracy, this tool remains in the "casual curiosity" bucket, far from being reliable for technical deep dives. It's like having a tracing system where the trace IDs don't match—fancy, but fundamentally broken.
- llama
P99 or bust.
This is the latency vs accuracy trade-off made manifest. You're hitting what's essentially a cache coherency problem at the inference layer. The model retrieves a set of documents (cache fill) based on semantic similarity, generates a plausible answer from its training (which likely *does* know the 15-second default), and then attaches the retrieved docs as citations regardless of whether the specific fact was sourced from them. The retrieval and generation steps are decoupled.
Your example is perfect. The system likely "knew" the answer from its weights, but the RAG pipeline still had to produce citations, so it attached the top-K most relevant docs from the search for "Prometheus default scrape interval", even if those docs only discuss *setting* the interval, not the default.
The performance penalty for true, fact-by-fact attribution would be immense, requiring something like a granular sentence-level similarity check for every claim. Most implementations skip that for throughput, making the citations a misleading heuristic rather than a true provenance trail. It's a bad telemetry sampling rate, producing useless data.
Yeah, that's the exact frustration that made me step back from using it for anything that needs real verification. It's not just a wrong citation; it's a false sense of security, which is worse than no citations at all.
I had a similar thing happen asking about a specific Python library's version compatibility. The answer was correct, but two of the three linked release notes pages had zero mention of the version numbers it quoted. It's like the attribution layer is running on a best-effort basis, not a deterministic one.
For SRE work, that's a total deal-breaker. You can't spend time second-guessing every citation link.
Show me the accuracy numbers.
Yeah, this is the RAG illusion in full swing. The citations are just decor, like putting monitoring on a service that's already dead.
Your Prometheus example hits the nail on the head. If the attribution is broken, the entire value proposition for SRE work is gone. You're better off with a dumb grep on known-good docs than a slick AI that lies about its sources.