Both. The JSON schema is your first and strongest guardrail to catch malformed output. But for a customer-facing summary, you also need content validation. We combine a schema with a keyword check.
The schema ensures structure. A Pydantic model that enforces fields like `period_start`, `period_end`, `top_incidents` as a list. It fails the run if the agent invents a field.
Then we run the generated narrative text through a second check against a list of required terms (e.g., "uptime", "response time") and banned terms (e.g., "I apologize", "I cannot", "as an AI"). Missing a required term triggers a rewrite. A banned term fails the job entirely.
It adds maybe 50ms, but it stops nonsense from going out the door.
Build once, deploy everywhere
I agree with the core premise about log grouping, but I think your distinction between deterministic tasks and narrative generation is the crucial architectural boundary. The mistake is letting the agent make *any* logical decision.
> treat the LLM like a templating engine with extra steps
This is exactly right, but I'd phrase it as a constrained text transformation. The "narrative structure" you build outside is a JSON schema. The agent's prompt becomes "Fill this schema with natural language based on this data." Its entire world is the input data and the output schema. There's no room for it to decide to group logs or calculate trends; that's all done before it's called.
The validation then isn't just a whitelist; it's a schema validation plus semantic checks against the source data. If the summary mentions a "spike in errors" but the source data shows no error delta, the run fails.
Data over dogma