The prevailing pattern of storing prompts in ad-hoc text files, chat histories, or—worse—solely in a vendor's cloud interface is an architectural anti-pattern. It creates immediate drift between the logic of your codebase and the "code" that instructs the AI models shaping it. If your infrastructure is defined as code, your prompts should be too, treated as a first-class, version-controlled artifact.
Consider this: a Terraform module that provisions an LLM-powered application component. The module's efficacy isn't just in its HCL; it's equally dependent on the precise system prompt used to generate or validate configurations. Storing that prompt elsewhere breaks the atomic commit. The solution is to co-locate prompts with the code they directly influence, using the same review and CI/CD gates.
A pragmatic structure for a project might look like this:
```
my-ai-service/
├── src/
├── infra/
│ └── main.tf
└── prompts/
├── code_generation/
│ └── api_handler_spec.md
├── code_review/
│ └── security_audit.md
└── validation/
└── terraform_plan_analyzer.md
```
Each `.md` file is a versioned prompt template, potentially with placeholder variables. Your orchestration scripts or CI jobs then inject context, much like a template engine.
```bash
# Example: Assemble a prompt for a CI review step
PROMPT_TEMPLATE=$(cat prompts/code_review/security_audit.md)
REPLACED_PROMPT=${PROMPT_TEMPLATE//"{{CODE_BLOCK}}"/"$(cat new_feature.py)"}
# Then call your AI provider API with $REPLACED_PROMPT
```
Key considerations for this approach:
* **Template Language:** Use a simple token replacement (`{{}}`), or for complex cases, integrate a lightweight engine like Jinja2 (Python) or Handlebars.
* **Testing:** Prompts must be tested. Create a test suite that runs known inputs through the prompt+model combination and validates outputs against expected patterns, much like unit tests for functions.
* **Secrets & Context:** Never store sensitive data in prompts. Use a context-fetching step in your pipeline to retrieve necessary secrets or code context dynamically and inject it safely.
* **Versioning the AI Model:** It is crucial to also version the *model identifier* (e.g., `gpt-4-1106-preview`) alongside the prompt. A prompt's behavior is a property of both its text and the model version.
The alternative—a separate "prompt repository"—introduces synchronization overhead and is only justified for organization-wide, generic prompt patterns. For project-specific instruction, tight coupling with the codebase is a feature, not a bug. It ensures reproducibility, enables rollbacks, and provides the necessary audit trail for when an AI-generated piece of code inevitably needs a root-cause analysis.
--from the trenches
infrastructure is code
I'm Cameron, principal engineer at a ~200 person fintech with a heavy serverless and container workload; we run a half dozen internal LLM applications in prod for code generation, compliance checks, and alert summarization, and we version all operational prompts directly in our main monorepo.
1. **Git-only vs. dedicated store**: Sticking prompts in your existing repo is zero-cost and gives you atomic commits, but it lacks any structured testing or environment-specific overrides. Dedicated platforms like PromptHub or PromptLayer cost $12-25/user/month and add API call tracking, but now you have drift risk between their cloud and your Git. We tried both; the dedicated tools become another vendor silo you have to reconcile.
2. **Test harness integration**: If you co-locate prompts in code, you must build your own validation. We use a simple pytest fixture that loads `.md` files and runs them through a validation schema (checks for unfilled placeholders, token length). Our CI blocks merges if a prompt change breaks validation, which takes about 4-5 seconds per prompt. A platform would give you this out of the box, but you pay in latency and vendor lock.
3. **Environment management headache**: The biggest hidden cost is managing staging vs. prod prompts. With Git, we use a trivial directory structure (`prompts/prod/`, `prompts/staging/`) and a config map to inject the path. It's brittle. Dedicated tools have environment flags, but then you're managing access controls and permissions in a separate system, which added roughly 2-3 hours a week of overhead for our team.
4. **Rollback and audit fidelity**: Git's rollback is perfect and instantaneous. When a prompt change degraded our code review agent's performance, we reverted the commit and redeployed the service in under 90 seconds. With a vendor, you're dependent on their versioning API, which in our test of one platform added a 30-45 second delay to fetch a historical prompt, not counting deployment time.
My pick is the boring Git-based approach for any team already practicing infrastructure-as-code, because the atomic commit is non-negotiable for us. If your team lacks the discipline to write prompt tests, or you need to share prompts across dozens of services with different release cycles, then a dedicated platform might be worth the tax - tell us your team size and how many distinct services touch prompts, and the call becomes obvious.
Trust but verify.
You're right about drift, but co-locating prompts as markdown files is a half-measure. It solves versioning, but not the actual problem: validation.
What's in your markdown file? A template with variables. How do you know it still works after your last refactor? How do you prevent a dev from "improving" a prompt for the staging environment and accidentally breaking the logic that production depends on? Your CI/CD gates are checking for a syntax error in a Terraform module, not for a regression in a prompt's expected output.
You've traded cloud vendor lock-in for a new kind of drift: between the prompt text and its operational correctness. Git won't tell you that.
Yeah, I've been trying that approach with a small pipeline at work. Having the prompt in a markdown file right next to my Python DAG is super clear for newbies like me.
But I'm already hitting a snag. How do you handle prompts that need slightly different wording for dev, staging, and prod? Like a prompt that summarizes data quality checks. I wouldn't want the dev version joking around in prod. Do you make three separate markdown files, or do you template it inside the code?
rookie
You've identified the core limitation precisely. Git tracks changes to the text, not to the model's behavior or the quality of its output.
The validation gap you describe is why we treat prompts as a specialized configuration file type. We run them through a pipeline that includes:
- Schema validation for template variables to catch missing interpolations.
- A small, deterministic test suite that calls the LLM with a frozen set of inputs and compares outputs using semantic similarity, not exact match, against known good baselines.
- Automated canary runs in a staging environment that flag significant deviations in output structure.
This moves the prompt from being a static document to a testable asset. The drift risk then shifts from "does the text match?" to "does the system still function as intended?", which is a more meaningful metric for CI/CD to enforce.
Migrate slow, validate fast.