Our CTO got a shiny new enterprise Perplexity license. Decided the interns should "leverage AI." They rolled it out to eighty of them.
First, the search history integration is a data leak waiting to happen. Interns pasting in internal error messages, snippets of our code, even AWS console output. Perplexity's now happily citing our internal architecture back to us, sourced from who-knows-where.
Second, the "pro" search results are useless for anything technical. They'd ask for a specific Postgres error code fix and get a generic blog summary from 2022. Ended up back on stackoverflow in five seconds. The interns figured out it's decent for summarizing public marketing pages. That's about it.
Biggest failure was assuming it could replace a basic internal wiki search. It can't. It hallucinates internal tool names and procedures. We spent more time correcting the AI's bad answers than if we'd just improved our documentation.
So we paid a premium to teach interns not to trust the AI. Money well spent, I suppose.
Your vendor is not your friend.
The data leak point is critical and often overlooked. Every vendor demo shows the public query examples. They never show what happens when someone pastes a `SECURITY_GROUP_ID` or a chunk of a staging database schema.
We had a similar scare with a Copilot rollout. The policy is now simple: any AI tool that can't be pointed *exclusively* at our own internal knowledge base, with a guaranteed airgap from its training data, gets blocked at the firewall.
For the search, you've hit on the real issue. These models are optimized for general web content. Your internal wiki is a tiny, specific corpus. A basic Elasticsearch instance with a decent synonym file would outperform it for 90% of internal queries, at a fraction of the cost.
Show me the query.
Yeah, the data leak thing is the quiet killer. People test with public info, but the second you let it near real work, internal schema and configs get slurped up. Good luck ever getting that data out of the model's training pipeline.
On the wiki search failure, that's a classic case of misapplied tech. These LLMs are essentially fancy, lossy compressors of the public web. Your internal docs are a weird little outlier distribution they've never seen. It's like using a sledgehammer to tap in a finishing nail.
I'd bet a basic BM25 search over your Confluence would have given those interns better results, and for free. The interns figuring out it's only good for marketing page summaries is the most damning review possible.
Absolutely. The "fancy, lossy compressors of the public web" is such a perfect description. It cuts right to why they fail internally.
I ran into this trying to use a similar tool for our product's API documentation. The model would confidently invent endpoints and parameters that didn't exist, stitching together concepts from other public APIs it had seen. It was creating a plausible-sounding but completely fictional version of our own product. That's when you realize these tools aren't searching, they're generating a "best guess" based on a totally different data universe.
Your BM25 suggestion is spot on. Sometimes the simplest, most boring tech is the right tool. We finally set up a proper internal search with synonyms for our jargon, and the difference was night and day. The interns probably would've built it themselves if given the chance instead of the shiny AI toy.
Happy testing!
You're glossing over the biggest cost. What's the Perplexity enterprise license running you, and what's the hourly burn for eighty interns futzing with it instead of using the wiki?
You got a screenshot of the billing breakdown? I bet the "pro" search credits are a separate line item that's already blown past the forecast.
The data leak is bad, but the financial leak is immediate. You paid for a sledgehammer and used it to miss the nail entirely. A decent Elasticsearch cluster for your internal docs would've cost a fraction for a whole year.
show me the bill