I've been testing fully automated pipelines for internal documentation and knowledge base articles. Goal: zero human touch after initial config.
Current test setup:
* Source: Confluence/Jira tickets -> formatted via webhook.
* Generator: Claude 3.5 Sonnet via API.
* Structuring: Custom Python parser creates strict XML templates.
* Publishing: Direct to internal CMS via REST.
Results from last 100-piece batch:
* Avg. factual accuracy (vs. source): ~92%
* Avg. readability score (Flesch): 68
* Pipeline runtime: ~4.2 minutes per piece
* Human override required: 7 pieces (due to ambiguous source)
Main failure points were in merging contradictory ticket updates. The system now flags low-confidence merges for review, but doesn't hold the pipeline.
Key config block for the content structure:
```xml
${step_one}
${step_two}
${resolution_one}
```
The rigid schema is what makes no-edit possible. Has anyone else run similar tests? Looking for data on how you measure acceptable quality thresholds for internal vs. external content.
- bench_beast
Benchmarks don't lie.
That ~92% factual accuracy is impressive for a zero-touch run. Our team tried something similar but stalled because we got scared about silently publishing wrong data. We just couldn't agree on an acceptable threshold.
For internal content, are you also measuring something like "time to correction"? Our worry was that a factual error in a process doc could cascade before anyone noticed. If your pipeline flags low-confidence merges but doesn't stop, how do you handle the cleanup? Is it a separate alert queue?