Pinning the CLI version with `requirements.txt` seems like a solid way to automate governance. My team is new to this though, and that rollback plan you mention makes me a bit nervous.
What does a "solid rollback plan" look like in practice for you? Is it just a quick git revert, or something more involved because of the pinned dependency? Asking because our last schema fix broke a few downstream tools for a couple of hours 😅
Just my two cents.
You're right to feel that department-based organization breaks down quickly. When we passed 50 workflows, we shifted entirely to a task-oriented taxonomy: generation, classification, transformation, extraction, and planning. This maps directly to the LLM's functional role, not our internal structure.
For reuse, we treat prompts as internal APIs. Each gets a unique identifier like `extraction.invoice_amount.v2` and a YAML metadata block declaring its contract. We use a simple registry service that maps these IDs to the current prompt text, allowing services to reference them by ID without needing the full template. This way, updating a shared prompt only requires a version bump in the registry, and you can gradually migrate consumers.
We did integrate with GitHub, but for metadata, not just the prompts. A Git pre-commit hook validates the metadata block - it checks for unique IDs and ensures any listed "consumer" services actually exist in our service directory. This static analysis catches integration errors before merge. For tags, we found the most valuable metadata was `risk_tier` (e.g., pii_handling, public_content) and `expected_input_schema`, which we validate at runtime.
What's your current process for detecting when a prompt change might break a dependent workflow you didn't know about?
Totally feel the jungle vibes. I went down a similar rabbit hole.
You gotta drop the department folders. They create silos and hide reuse. I switched to organizing by **AI task type** (generation, classification, etc.) and it clicked. A support ticket classifier and a marketing email classifier are the same *type* of work, even if the content differs. Lets you compare and improve patterns.
For reuse, treat prompts like microservices. Give each a unique ID (e.g., `generation.email_variant.v2`) and a central registry. Services call the ID, not the raw text. Lets you update once and track dependencies.
My clever metadata hack? Tag by **cost** (high/med/low volume) and **risk** (handles PII? public facing?). Helps prioritize reviews and catch changes that could blow up your bill or compliance.
Demo or it didn't happen
Love the `generation.email_variant.v2` naming scheme, it's almost like a package manager. We did something similar but added a prefix for the model (`gpt-4o-generation.email_variant.v2`) because we found cost and behavior could shift dramatically between models, even for the same task type.
Your cost/risk tagging is spot on, but it made me wonder - how do you handle prompts that *change* risk tier over time? We had a simple summarizer get adapted to handle customer names, silently moving from low to high risk. We ended up adding a required `audit_log` field in the metadata to flag when those core tags are edited.
Curious, do you tie your risk tags to any automated compliance checks, or is it purely for human review?
Still looking for the perfect one
That's a fascinating complication, adding the model prefix. I hadn't considered that, but you're right, the behavioral drift between models for the same prompt logic can be significant. Does that ever create a maintenance burden for you when a new model version rolls out, requiring you to essentially duplicate and test prompts across model-specific files?
On your question about automated checks for risk tags, we're just starting to explore that. It's mostly human review now, but we're piloting a script that scans the `pii_handling` tagged prompts against a list of known sensitive data patterns in the prompt text itself. It's not perfect, but it flags mismatches for a second look. Your `audit_log` field is clever, though. How do you enforce that it's actually filled out, is it another hook?