Right there with you. That YAML test is the perfect filter. Even if it passes syntax, I'd be worried about the actual thresholds and labels.
I built a small test harness last week for generating Kubernetes manifests. The real killer wasn't indentation (though that happened). It was subtle deviations in label selectors or security context fields that would pass `kubectl apply --dry-run` but cause the pod to sit in Pending. You're spot on: if it can't handle the boring stuff, it's just a toy.
Maybe the first "plugin" should just be a linter/validator that runs after generation, using the same spec you'd give a human. But then... you're back to writing the spec properly in the first place.
Clean code, happy life
The YAML test is the right starting point, but you need to go a step further. Even if the syntax is perfect, the operational logic has to be correct for your stack.
I tried your exact prompt through Workato last year with another model API to generate CloudWatch alarms. The YAML was valid. The alert name was `HighMemoryUtilization`. Problem was, our metric is called `MemoryPercent`, not `MemoryUtilization`. The alert never fired. The API gave me a perfectly formed brick.
So your plugin now needs a mapping layer between generic terms and your actual observability schema. At that point, you're just building a bad templating system with a stochastic engine in the middle.
Integration is not a project, it's a lifestyle.
Exactly. That's the trap. The API gives you a correct generic answer, but your environment isn't generic.
Adding a mapping layer just creates a new failure mode. Now you have to maintain a perfect ontology of your own metrics, and you're still one hallucination away from a silent failure. It's cheaper and more reliable to write the three-line template.
Good point about needing a real test. I think the YAML check is a great start.
So if the test passes, what's the next boring task you'd check? Something from a real SaaS setup, maybe generating a basic lead scoring rule in a CRM? I'm wondering what the simplest production use case would be.