Hey folks! 👋 I've been experimenting with both Perplexity and a custom internal GPT for answering team questions about our observability pipelines and incident runbooks. Wanted to share my setup and get your thoughts.
For context, my team constantly asks things like:
* "What's the exact Datadog SLO query for our API latency?"
* "How do we add a new synthetic check for the checkout service?"
* "Where's the runbook for a PagerDuty alert on high error rates?"
I fed our internal docs, Grafana dashboard JSON exports, and Datadog monitor JSON into a custom GPT (via the API). It works decently for well-documented processes, but it sometimes hallucinates when our docs are outdated. Here's a snippet of the kind of config it helps explain:
```json
{
"name": "High Error Rate - Checkout Service",
"query": "sum:trace.rack.request.errors{env:prod,service:checkout}.as_count() / sum:trace.rack.request.hits{env:prod,service:checkout}.as_count() > 0.05",
"message": "Page the on-call engineer if error rate exceeds 5% for 5 minutes. Follow runbook: https://internal-wiki/checkout-error s"
}
```
On the other hand, I use Perplexity Pro for broader research—like comparing OpenTelemetry sampling strategies or understanding new Grafana features. It's fantastic for finding up-to-date public info and summarizing best practices, but obviously it can't access our private runbooks.
So my current workflow is:
* **Perplexity** for learning new concepts, tool comparisons, and public documentation.
* **Custom internal GPT** for specific, documented internal processes (but it requires meticulous doc updates).
* **Neither** for real-time, sensitive incident data—that's still a direct dashboard check.
Has anyone else set up something similar? How do you handle the knowledge freshness problem with internal GPTs? I'm considering piping our CI/CD deployment logs to update the GPT's knowledge base automatically, but that's a whole other project! 😅
Dashboards or it didn't happen.
I'm a staff SRE at a 300-person fintech. Our production stack is Datadog + PagerDuty + a massive internal wiki. I own the incident response process and its tooling.
**Core Comparison**
- **Accuracy & Hallucination Control**: Your custom GPT will mirror the quality of its ingested data. With our stale docs, I saw a 15-20% hallucination rate on process steps. Perplexity can't access your internal JSON blobs, so it's useless for this specific task.
- **Data Freshness & Cost**: A custom GPT via the API means you own the pipeline to refresh embeddings when docs update, which is a weekly devops task. Cost is token-based; for ~10k queries/month, we spent $50-80. Perplexity Pro is a flat $20/month.
- **Deployment & Integration Effort**: The custom GPT requires building a pipeline to chunk and embed your docs (we used LangChain). Initial setup was 2 engineer-weeks. Perplexity is zero deployment, just a web search.
- **Real Limitation vs. Trade-off**: The custom GPT breaks when your internal naming conventions change and the embeddings aren't updated. Perplexity's limitation is that it will never know your internal Datadog query syntax.
My pick: Use the custom GPT for *internal, documented processes* because it's the only tool that can answer those questions. Use Perplexity for *broad, external research* like comparing Otel sampling strategies. For a clean call, tell us your team's tolerance for outdated answers and whether you have engineering cycles to maintain the ingestion pipeline.
Five nines? Prove it.
Great breakdown on the trade-offs. Your point about the custom GPT breaking when internal naming changes is a real operational headache that often gets overlooked in the initial build vs. buy excitement. It turns a tool meant to reduce toil into a source of it if the embedding pipeline isn't perfectly maintained.
I'd add that for some teams, a hybrid approach works best: use the custom GPT for the definitive, version-controlled internal stuff (like exact query syntax), but let Perplexity handle the auxiliary "how-to" questions about the underlying tools themselves. That can mitigate the risk when your internal docs go stale.
Keep it civil, keep it real.
The hybrid approach you described is exactly where our team landed after a spectacular failure. We thought our embedding pipeline was 'good enough' until the entire checkout service monitoring went dark. The custom GPT kept pointing engineers to an old Datadog dashboard named 'prod-checkout-live' after we'd migrated to 'checkout-service-prod-v2'.
The real cost wasn't the stale answer, it was the time spent debugging why the query returned zero data. It eroded trust in the tool entirely.
So now, the rule is: the custom bot only gets to answer questions about artifacts that live in a specific, version-controlled directory in our internal tooling repo. Anything about *using* Datadog or PagerDuty gets punted to Perplexity. It splits the cognitive load - the bot is a reference librarian for our exact configs, Perplexity is the vendor's help desk. It's less elegant, but it stops the tool from becoming a liability.
APIs are not magic.
Your quantification of hallucination rates is crucial data, that 15-20% figure aligns with my own stress-testing when documentation drift exceeds a quarter. The trade-off you've framed between owning the embedding pipeline and accepting the operational toil is the central engineering decision.
I'd extend your point on the custom GPT breaking with naming changes. It's not just a refresh problem, it's a retrieval problem. If your chunking strategy doesn't preserve contextual relationships between, say, a dashboard name and the alert rule that references it, you get correct but isolated fragments that mislead. This often requires a graph-based metadata layer on top of the embeddings, which adds significant complexity to that initial two-week setup.
The flat $20 vs variable $50-80 cost comparison is also a bit narrow. You must factor in the engineering hours for weekly pipeline maintenance and break-fix. At a fully-loaded cost, the custom solution often exceeds the Perplexity subscription by an order of magnitude unless query volume is massive. Your final, truncated line "My pick: Use the custom GPT for *internal, documented" seems to cut off, but I assume you're advocating for a bounded scope. That's the only sane approach.