Yeah, the "false sense of compliance" is the real kicker. It *looks* right at a glance. I've had the same thing with internal CSS utility classes - it'll generate something that follows BEM's *structure* but uses our deprecated naming prefix, and a quick review misses it because the pattern *feels* correct.
Your point about it being semantic, not syntactic, failure is spot on. The linter passes it, but the integration test fails hours later. Makes me think we need to prompt for the *why* behind the rule, not just the rule text. Like "prefix with `ACCT_` because the event bus router filters on it". Sometimes that context sticks better than the config.
Prompt engineering is the new debugging
The "machine-readable spec" point is exactly why I gave up on pasting configs for cost policy generation. It'd ignore our internal tagging format even after seeing the exact regex pattern.
I've had some luck with a different trick, though. Instead of the config, I paste a few lines of *correct* SQL from our codebase right before the request, as a concrete example of the style. Something about seeing the pattern in executed code seems to anchor it better than the abstract rules file. Still a coin toss on complex joins, but it improved my baseline.
That's a really interesting workaround. Using concrete examples over abstract rules makes sense from a statistical learning angle, right? The model sees an actual valid token sequence it can mimic, rather than trying to interpret a rule and generate something that fits it.
But does that scale? If I have to paste example SQL for every different type of query pattern, that's a lot of prep work. I wonder if you could prime it once in a session by showing a few diverse "correct" examples upfront, then ask for new variations. Have you tried that, or is it always a per-request thing?
That priming strategy is exactly what I've been testing, and the persistence is the real issue. You can provide a diverse set of examples at the start of a session and get compliance for the first few requests. But in my experience, the model's internal priors reassert themselves after about three or four generations, especially when you switch to a slightly different but related task. It's like the initial examples create a temporary, fragile context that gets diluted.
So it scales poorly. You're not just doing the prep work once; you're fighting a decaying attention window. I've resorted to a hybrid approach where I include a single, relevant example *inline* with each new request, treating it as a necessary cost. Even then, you need to be precise. A `SELECT` example won't anchor a `CREATE TABLE` statement unless the naming convention is the *only* rule, because the model treats the structural difference as a separate domain.
It points to a fundamental mismatch: we're treating it as a reasoning agent that understands rules, but it's a statistical sampler with a very strong bias towards its training distribution. The "workaround" is really just manipulating that sampling space with more weighted examples, which is inherently labor-intensive.
Trust but verify.