Used it daily for a month to generate IaC configs, summarize incident post-mortems, and get quick tech stack comparisons. The time saved is real. I'd estimate 15-20 hours.
But the convenience has a tax. Found three critical errors in Terraform code it generated for a simple GCP networking module. It hallucinated attribute names that don't exist in the provider.
```hcl
# Perplexity suggested this
resource "google_compute_firewall" "rule" {
name = "allow-ssh"
network = google_compute_network.vpc.name
allow {
protocol = "tcp"
ports = ["22"]
}
source_ranges = ["0.0.0.0/0"]
direction = "INGRESS" # This 'direction' argument doesn't exist for this resource
}
```
Had to catch it myself. The summaries are slick, but they smooth over crucial technical nuance. It's a fantastic over-glorified search aggregator that's increasingly confident about being wrong. Saved time on the front end, now I triple-check everything. Not a net win.
Keep it simple
Your example with the Terraform `direction` argument is a perfect illustration of the core problem. These models are trained on a corpus of public code, and for a resource like `google_compute_firewall`, that includes both the older `google-beta` provider syntax and the current GA provider. It's likely blending them without version awareness.
I've hit a similar issue generating BigQuery DDL. It will confidently produce syntax from a legacy version of Standard SQL, like an outdated `CREATE MODEL` statement, which fails silently if you don't catch the version mismatch. The time saved on drafting is immediately consumed by a more insidious validation step - you're not just checking for logic, you're auditing for anachronisms.
That's why I've settled on a strict rule: these tools are only for generating first drafts of *non-executable* artifacts. Summaries, comment blocks, boilerplate documentation. The moment it produces executable code - SQL, Terraform, Python - the cognitive load to verify it exceeds just writing it from a known-good template or the official docs. The "net win" disappears when the error could be a subtle data type mismatch that surfaces in production.
data is the product
Totally feel this. Your point about auditing for anachronisms hits home - it's like having a coworker who's brilliant but stuck three years in the past.
I've found a slightly different middle ground for executable code. I'll use it to generate a snippet, but then I immediately feed that output into a secondary prompt like: "Review this [language] code for deprecated arguments or syntax mismatches with [specific library] version [X.Y.Z]." The model can sometimes catch its own anachronisms when asked to play a different role. It's not perfect, but it adds a useful filter.
That said, your rule about sticking to non-executable artifacts is probably the saner default. The mental tax of verifying is real.
Prompt engineering is the new debugging
Yep, that `direction` argument is a classic. It's from the `google_compute_firewall_policy_rule` resource, not the standard firewall. The model mashed them together.
My rule now: it writes the first draft, but the linter and `terraform plan` are the final reviewers. I caught a similar bug where it used a deprecated `gcp_` prefix for a service account key. It would have blown up our deployment at 3am.
The time save is real for boilerplate, but you're right, the trust is gone. It's like getting a beautifully formatted runbook from the last person who left. You still have to dry-run it yourself before the pagers go off.
Pager duty is not a hobby
Totally get that feeling of trading trust for time. Your mention of summaries smoothing over nuance is spot on, especially for incident post-mortems. I've had it draft a root cause analysis that conveniently omitted a key service dependency, making the timeline look much cleaner than reality. Great for a first draft, but dangerously coherent.
For IaC, my team landed on a rule: treat any generated config as un-reviewed pseudocode. It goes straight into a CI pipeline that runs `terraform validate` and a provider-specific linter before a human even looks at it. The speed is still there for drafting, but we've automated the "triple-check" into the system.
It's becoming less of a research tool and more of a very fast, occasionally creative, intern who needs constant supervision.
Your example hits on the exact workflow gap I've been trying to bridge. The time save is real, but the verification tax kills it.
I've started treating these tools as the "generate" step in a custom n8n chain. It spits out the code, then an automation immediately fires it against the actual provider's API schema - or runs `terraform fmt` and `validate` in a container - before it even lands in my editor. The LLM's output becomes just another unreliable data source that needs plumbing into a sanity-check system.
It turns the "triple-check everything" from a mental burden into a scheduled job. You keep the 15-20 hours and hopefully avoid the 3am blowup.
if it's manual, it's wrong