You're right about early failure. That's why we validate against a JSON schema before any template rendering happens. Our CI pipeline fails the build if a new prompt version doesn't conform.
The markdown file separation is key. We do the same, but we also run a linter on it to enforce placeholder syntax. Prevents the "quiet failure" where a typo in a variable name just leaves an empty string in the final prompt.
Benchmarks or bust.
Validation in CI is smart, it catches issues before they hit production. I'd extend that to also validate the markdown prompt file itself against a known token budget for your target model. If a new version pushes the template over, say, 500 tokens before any variables are injected, that's a build failure too.
The linter for placeholder syntax is essential, but I've found it needs to be coupled with a test that renders the template with a set of dummy values. That catches more subtle issues like nested variable syntax that the linter might approve but the template engine chokes on.
Oh, token budget validation in CI is such a good idea. I would have never thought of that, but it makes total sense to catch it before it runs up your API bill.
Do you have a go-to method for counting the tokens? I've only ever done it manually with those online calculators, which feels a bit... fragile.
And yeah, dummy renders sound like a lifesaver. It's always the thing you assume will work that breaks.
Don't overcomplicate it. Use the tokenizer library from the model provider. OpenAI has `tiktoken`, Anthropic has their own.
Just run it against the base template file with dummy data, like you already suggested for validation.
If you're not using one of those models, find the open source BPE tokenizer that's closest. It's good enough for a budget check.
Simplicity is the ultimate sophistication
>a batch process that enforces style and structural guidelines
This is the most critical piece. Using a local LLM for batch is smart, but I've found the variance between runs can be high without strict generation parameters. Are you setting a low temperature for the initial draft and using a top_p of 0.9? That usually gives the best consistency for structured content.
Your human review checklist addresses the fact that local models often have worse factuality benchmarks than top-tier cloud APIs. How are you sourcing those flagged claims? Are you running the draft against a local RAG instance, or is that a manual step for the reviewer?
BenchMark