How do I set up a repeatable content generation pipeline with AI writing tools?

The promise of "AI-driven content pipelines" is often drowned in vendor hype and vague case studies. Let's cut through that. A truly repeatable pipeline is not about a single magical tool; it's a disciplined, multi-stage workflow that treats AI output as a raw material to be processed, validated, and refined. The goal is consistent, high-quality output with predictable costs and minimal last-minute human firefighting.

Based on implementing this across several engineering blogs and documentation projects, the core architecture separates **Generation**, **Validation**, and **Publication** into distinct, automatable phases. Here is a functional blueprint.

### 1. Generation & Structuring
This phase is about creating structured briefs and initial drafts. Never prompt an AI with "write a blog post about X." You must provide a detailed template.

```yaml
# content_brief_template.yaml
topic: "Kubernetes Horizontal Pod Autoscaling with Custom Metrics"
target_audience: "Platform Engineers with intermediate k8s knowledge"
primary_keyword: "kubernetes custom metrics autoscaling"
secondary_keywords: ["Prometheus Adapter", "HPA", "External Metrics"]
content_type: "Tutorial"
word_count_target: 1500
structure:
- "Introduction: Problem of scaling on CPU/Memory alone"
- "Prerequisites: k8s cluster, Prometheus, metrics server"
- "Architecture: Diagram of Prometheus -> Adapter -> HPA"
- "Implementation: Code blocks for installing Prometheus Adapter"
- "Configuration: Example HPA manifest with `metrics` field"
- "Validation: How to test and verify scaling is working"
- "Cost Considerations: Caveats on metric aggregation intervals"
- "Conclusion: Summary and links to official docs"
tone: "Technical, precise, avoid marketing fluff"
sources: ["Official Kubernetes HPA docs", "Prometheus Adapter GitHub README"]
```

This brief is then fed to your primary LLM (e.g., via the OpenAI API, Anthropic, or a local model). The output is a first draft.

### 2. Validation & Enrichment
The raw draft is the weakest link. It requires automated checks before any human sees it. A simple script can run these validations:
* **Fact/Code Check:** Run code blocks through a syntax linter (e.g., `shellcheck` for bash, `prettier --check` for YAML/JSON).
* **SEO & Readability:** Use tools like `write-good` for passive voice, check keyword density is within a target range.
* **Plagiarism & Originality:** Run a sentence-level similarity check against a known source corpus (this is critical for maintaining domain authority).
* **Internal Linking:** Check against a sitemap for suggested links to existing content.

### 3. Human-in-the-Loop (The Critical Stage)
The validated draft moves to a human editor via a platform like Google Docs or a CMS with review workflows. The editor has a clear checklist:
* [ ] Technical accuracy of all claims and steps.
* [ ] Code examples are functional and follow our internal standards.
* [ ] Narrative flow is logical and matches the target audience.
* [ ] All sources are correctly cited; no "hallucinated" references.
* [ ] Tone is consistent with our brand voice (the AI's biggest weakness).

Edits are made directly in the document. For the next iteration, *this edited version becomes the new gold-standard source* for fine-tuning or providing as a better example to the LLM, creating a feedback loop that improves the entire pipeline.

### 4. Publication & Orchestration
The final, approved content is pushed via API to your CMS (e.g., WordPress, Ghost, Hugo static site). This step should be fully automated from a Git repository or a dedicated publishing tool. The entire pipeline can be orchestrated with a tool like **Apache Airflow** or even a series of **GitHub Actions**:

```yaml
# .github/workflows/content-pipeline.yml
name: Content Pipeline
on:
workflow_dispatch:
inputs:
brief_path:
description: 'Path to content brief YAML'
required: true

jobs:
generate:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Generate Draft
run: |
python scripts/generate_from_brief.py ${{ github.event.inputs.brief_path }}
validate:
needs: generate
runs-on: ubuntu-latest
steps:
- name: Lint Code Blocks
run: scripts/lint_code.sh
- name: SEO & Readability Check
run: scripts/quality_checks.sh
create-pr:
needs: validate
runs-on: ubuntu-latest
steps:
- name: Create Editorial Pull Request
run: |
# Creates a PR with the draft, tagging the editor team
```

**Key Metrics to Monitor:** Cost per article (API token usage), average human editing time, readability score trend, and content performance (traffic, engagement). If your editing time isn't decreasing over time, your validation or briefing stage is failing.

The tools are interchangeable; the rigor of the process is what creates repeatability. Without the validation and strict editorial gate, you are merely automating the production of low-quality, potentially inaccurate text.

-- alex