I've been testing You.com's developer-focused features for generating code snippets and explaining APIs. The speed is there, but I've noticed a critical flaw: the citations to official documentation are often wrong.
For example, I asked for a Salesforce Bulk API 2.0 example to create jobs. The provided code snippet was cited as coming from "Salesforce Developer Documentation, Bulk API Guide." When I followed the link, it redirected to a generic REST API overview page, not the Bulk API 2.0 specifics. The actual parameter structure in the example was from an older API version. This isn't just inconvenient; it breaks trust. If I'm evaluating this for a team, I can't have junior developers wasting hours debugging based on a hallucinated citation.
Has anyone else run reproducibility tests on this? My process:
1. Ask for a technical snippet with a specific framework/API version.
2. Check every linked citation. Does it land on the exact page, and does the text around the anchor match the claim?
3. Attempt to run the code against the actual platform's sandbox.
My failure rate on step 2 is over 50% for Salesforce, HubSpot, and Stripe docs. The code sometimes works, but the citation is a mirage. This makes it useless for audit trails or verifying best practices.
Are there specific domains (e.g., AWS vs. Azure) where you're seeing better accuracy, or is this a universal problem with their retrieval? I'm leaning towards this being a dealbreaker for any serious technical workflow until they publish their grounding methodology and show measurable improvement.
Show me the query.
Totally agree on the core issue. I've hit similar problems with Kubernetes CRD documentation citations. It'll cite the exact `apiVersion` and field, but the link goes to the main Kubernetes API concepts page, not the specific resource.
Your reproducibility test is smart. I've started treating these citations like a Helm chart - always verify the `values.yaml` against the actual template output. The code might deploy, but if the source is wrong, you're one upgrade away from a breaking change.
Have you found any pattern? For me, it seems worse with rapidly versioned APIs (K8s, Terraform providers) versus more stable ones.
#k8s
That's a really useful analogy, treating citations like Helm values that need verification. I think your observation about rapidly versioned APIs is key.
I've seen this in SaaS procurement contexts too, where an AI tool will cite a vendor's SLA from a specific date, but the link goes to their current marketing page. The terms, especially around data privacy or uptime, can shift subtly between versions. It creates a hidden compliance risk if you're basing a contract review on an outdated reference.
It makes me wonder if the issue isn't just the rate of change, but how these tools map a specific technical query to a URL. They might be grabbing the correct *text* from a source but assigning it the root URL of the documentation domain instead of the precise anchor.
read the contract