Alright, let's cut through the hype. Everyone's talking about AI assistants like they're magic, but we're engineers. We measure things. I've been evaluating Perplexity alongside other tools, not for buzzwords but for concrete, deployable answers in our domain—Kubernetes, observability, infrastructure.
I designed a controlled test: 10 technical queries, ranging from basic `kubectl` commands to complex Istio telemetry configurations. Each query was submitted to Perplexity (Pro, using the "Precise" focus), ChatGPT-4, and Claude 3 Opus. The goal was to assess accuracy, depth, and, critically, *actionability* of the output. No poetry, no marketing fluff—just code and configs.
Here are the summarized results, presented as a sort of `kubectl top pods` for LLMs. The key metrics were:
* **Correctness:** Was the core answer technically accurate?
* **Depth:** Did it go beyond a one-line answer? Did it include caveats (e.g., "this requires the `net-admin` capability")?
* **Actionability:** Did it provide a ready-to-use YAML snippet, command, or configuration block?
* **Hallucination Rate:** Did it invent non-existent flags or APIs?
| Query # | Topic | Perplexity | ChatGPT-4 | Claude 3 | Notes |
| :--- | :--- | :--- | :--- | :--- | :--- |
| 1 | `kubectl` to list pods sorted by memory | ✅✅✅ | ✅✅ | ✅✅✅ | All correct. Perplexity & Claude added `--sort-by` flag explanation. |
| 2 | Helm template for a ConfigMap | ✅✅ | ✅✅✅ | ✅✅ | ChatGPT's was most detailed with `range` loops. Perplexity was functional but bare. |
| 3 | Istio `PeerAuthentication` strict mode YAML | ✅✅✅ | ✅✅✅ | ✅ | Claude omitted the `portLevelMtls` example. |
| 4 | Prometheus rule for pod restart alert | ✅✅ | ✅✅✅ | ✅✅ | Perplexity's rule was correct but lacked the critical `for: 5m` stability clause. |
| 5 | Patch a Deployment to add a sidecar | ✅ | ✅✅✅ | ✅✅ | Perplexity's `kubectl patch` example was risky, used `merge` strategy incorrectly. |
| 6 | HPA YAML for custom Prometheus metric | ✅✅ | ✅✅ | ❌ | Claude hallucinated a non-existent `metrics` API field. Perplexity cited the correct `external.metrics.k8s.io`. |
| 7 | `kube-prometheus-stack` upgrade path | ✅✅✅ | ✅✅ | ✅ | Perplexity excelled here with specific Helm repo commands & version pinning. |
| 8 | Cost-optimized nodeAffinity for spot instances | ✅✅ | ✅ | ✅✅✅ | Perplexity provided good GKE/AWS examples. Claude had the most robust tolerations. |
| 9 | Grafana Loki log filter query for 5xx errors | ✅ | ✅✅✅ | ✅✅ | Perplexity's LogQL was basic, missing label filters. |
| 10 | Migrate `PodSecurityPolicy` to Kyverno | ✅✅ | ❌ | ✅✅✅ | ChatGPT still recommended the deprecated PSP. Perplexity gave a valid Kyverno `ClusterPolicy`. |
**Analysis & Key Takeaway:**
Perplexity operates like a competent, fast `initContainer`. It gets the job done for well-trodden paths with remarkable speed and solid citation, which is valuable. However, for complex, production-grade "deployments"—where nuance, edge cases, and safety matter—it sometimes lacks the depth of its peers. Its strength is in retrieval and synthesis of current docs (the Istio example was flawless). Its weakness is in complex, multi-step architectural reasoning (the patch example was concerning).
For this community, my recommendation is:
* Use Perplexity as a first-line, high-speed search for **current API references** and **common snippet generation**. It's excellent for "What's the exact YAML for a Kubernetes `NetworkPolicy`?".
* Do not rely on it alone for **complex migrations**, **safety-critical patches**, or **deeply nuanced SRE policies**. For those, a more iterative chat with a model like Claude, or a specialist tool, is still required.
* The citations are its killer feature for validation, but always `kubectl apply --dry-run=client` its suggestions.
The full query set and raw outputs are in this Gist [link]. I've included the exact prompts and the generated YAML. Feel free to replicate.
-- k8s
That's a solid approach, and I'm keen to see the full results. The structure is good, but I'd suggest adding a fifth metric for these infrastructure tasks: **Currency.**
A command like `kubectl logs` might be accurate and actionable, but if it doesn't mention the `--all-containers` flag (which is relatively new), it's giving an outdated best practice. Did any of the models reference recent API deprecations or note that a solution was only viable from Kubernetes 1.27 onward? That's often the differentiator between a correct answer and a useful one in our field.
Can you share the raw queries? I'm curious if any touched on newer service meshes like Linkerd or Cilium.
Mike D.
Good on you for starting with measurement. I'm looking at your metrics and one's missing: latency. For pipeline work, a 100% correct answer that takes 45 seconds to generate is less useful than a 90% correct one in 5 seconds. Did you track response times?
Also, the hallucination rate is key. I've seen models invent entire `kubectl` subcommands. Post the raw data - I want to see which specific Istio telemetry config they botched.
shift left or go home
Love the "kubectl top pods for LLMs" framing. Makes it real.
Can you share the actual table? I'm especially curious about the Istio config answers. I tried a similar test last month with Terraform AWS security group rules, and Perplexity kept suggesting deprecated `egress` blocks. That actionability metric is key - a perfect YAML that uses an old API version is worse than a partial answer.