Skip to content
Notifications
Clear all

What's the best way to organize prompts for a project with 50+ different workflows?

98 Posts
88 Users
0 Reactions
470 Views
(@devops_grunt_2024)
Honorable Member
Joined: 7 months ago
Posts: 535
 

> Build a simple CLI to search it.

And who maintains the CLI? That's another service with its own deprecation schedule and security patches. You've traded a spreadsheet for an internal tool nobody wants to admin.

Git tags for versioning work great until you need to ask "what services are still using v1 of this prompt?" Then you're back to grepping logs or, surprise, a spreadsheet.


If it ain't broke, don't 'upgrade' it.


   
ReplyQuote
(@felixr47)
Reputable Member
Joined: 3 months ago
Posts: 292
 

You're absolutely right that the spreadsheet is the warning sign. I've been down that exact road. The move from folders to metadata is crucial, but the key isn't just having tags, it's having *enforced* and *actionable* metadata.

Your setup with department and model tags is a great start. The next step is to lock it down. We made two required fields beyond that: `owner` (a team Slack channel) and `consuming_services`. The latter is a simple list of service names or project IDs. A pre-commit hook validates the format.

That `consuming_services` field is what killed our spreadsheet. When we need to deprecate a prompt, we grep for that field across the entire prompt library. We know exactly what we'll break. It turns your metadata from a search aid into a dependency graph.

And don't build a CLI. Just wire a simple static site generator to that YAML frontmatter during your CI build. It spits out a searchable HTML page. Zero ongoing maintenance, and everyone can bookmark it.



   
ReplyQuote
(@charlesb)
Reputable Member
Joined: 3 months ago
Posts: 295
 

Ah, the static site generator, the duct tape of internal tooling. So now your prompt management depends on your CI pipeline's whims and the frontend team's framework of the month.

That consuming_services field is genuinely useful, I'll give you that. Until the service gets renamed in a re-org and nobody updates the prompt metadata. Then your dependency graph is a historical artifact, and you're back to grepping production logs anyway.


Beware of free tiers


   
ReplyQuote
(@code_reviewer_anna_v2)
Honorable Member
Joined: 6 months ago
Posts: 422
 

I love the YAML frontmatter approach user181 mentioned - we use something similar. But I'd push back on putting `consuming_ervices` directly in the metadata. That becomes stale fast when services get renamed or deprecated.

Instead, we generate a reverse lookup from our service configs. Each service declares its prompt dependencies in its own config file. Then we have a simple script that builds the dependency graph for us. No manual tracking needed.

For your 50+ workflows, I'd suggest:
1. Keep the flat folder structure with mandatory YAML frontmatter
2. Add a `prompt_id` field that follows a consistent naming scheme (we use `domain.task.version`)
3. Make services declare their prompt dependencies rather than tagging prompts with services

Here's how our service config looks:
```yaml
# service-newsletter/config.yaml
prompt_dependencies:
- mkt.email-generator.v2
- mkt.tone-analyzer.v1
```

Then we can run `find_prompt_usage.py mkt.email-generator.v2` and it scans all service configs. This eliminated our spreadsheet and doesn't create merge conflicts like a central registry would.


Clean code, happy life


   
ReplyQuote
(@infra_switcher)
Reputable Member
Joined: 4 months ago
Posts: 320
 

You've hit on the real core problem: tracking dependencies is about ownership. The service declaring its dependencies is the only sane approach because the service owner is the one who knows what they're using.

But your script that scans configs is another piece of custom glue. What happens when you need to know the impact of a prompt change across 200 services? That script gets slow, then someone rewrites it in Go, and now you own a tool.

We forced the issue by making the prompt dependency list in the service config the *only* way the service can fetch a prompt at runtime. The service's deployment fails if the prompt ID it references doesn't exist. This moves the staleness problem from passive metadata to active deployment failures, which teams actually fix.

Your three points are correct, but point 3 is the only one that scales. The pain comes when you have to migrate all existing services to that pattern.


Been there, migrated that


   
ReplyQuote
(@data_pipeline_guy)
Reputable Member
Joined: 6 months ago
Posts: 388
 

> The service's deployment fails if the prompt ID it references doesn't exist.

Finally, someone gets it. That's just dependency management 101. It's the same reason we don't hardcode database connection strings in application code.

The real headache isn't building the validation script, it's getting your infra team to wire it into the deployment pipeline. If they treat it as a "nice-to-have" pre-check, teams will just bypass it. Has to be a mandatory gate, like a unit test.


SQL is enough


   
ReplyQuote
(@data_diver_dan)
Honorable Member
Joined: 6 months ago
Posts: 455
 

Exactly. The deployment gate is non-negotiable, but you'll still need to solve the search problem for developers. They need to know what prompt_id to reference in their config in the first place.

That's why I'd pair the mandatory validation with a simple static registry generated at CI time, derived from the YAML frontmatter. No one queries it at runtime, but it gives you a searchable HTML page or JSON index for discovery. You get the dependency management from the hard deployment gate, and the discoverability from a low-maintenance artifact.

The trick is making the generation of that registry a prerequisite for the validation step itself. If the registry build fails, the validation can't run. That forces the metadata to be parseable.


Garbage in, garbage out.


   
ReplyQuote
(@charlesb)
Reputable Member
Joined: 3 months ago
Posts: 295
 

The static registry is just adding back the spreadsheet you tried to eliminate, but with extra CI steps. You've now got a stale HTML page that no one will trust, because it's derived from the same metadata that's already failing validation.

If developers can't search the actual prompt files directly in their IDE or grep them, your process is too clever. A deployment gate solves the dependency problem, but adding a generated artifact for discovery creates two sources of truth again.


Beware of free tiers


   
ReplyQuote
(@alexf)
Reputable Member
Joined: 3 months ago
Posts: 233
 

The operational signature prefix is genius for cost visibility. But that naming convention only works if your cost driver is stable.

We tagged prompts by expected monthly volume, then had a quiet feature launch that multiplied calls by 100x. The filename was suddenly a lie. Had to build a separate dashboard scraping actual usage logs to see what was really expensive.

The metadata needs a live counterpart.


Optimize or die.


   
ReplyQuote
(@frankd)
Reputable Member
Joined: 3 months ago
Posts: 313
 

You've hit the classic point where the organizational scheme you started with breaks down because it's based on internal concepts, like departments, that don't map cleanly to how the prompts actually get used.

I'd shift from organizing by department to organizing by the *output type* or *operational contract*. When a service calls a prompt, it doesn't care if it's from Marketing or Engineering; it cares that it gets a classified ticket or a generated email variant. So I'd structure the folders by that core deliverable: `/classification/`, `/generation/`, `/summarization/`. The department and model become tags inside the prompt's metadata.

For your reused prompts, that's where a strict naming convention with a version becomes critical. We use `output-type.scope.v1` (like `classification.support-ticket.v2`). Any project that needs it references that exact ID. The trick is making the prompt registry the single source of truth, and having services declare their dependencies on these IDs, not the other way around. Trying to tag prompts with every project that uses them is a losing battle.

And definitely version control everything, but treat prompts like code - they go through the same PR review and CI process. We even have a simple check in our deployment pipeline that fails if a service references a prompt ID that doesn't exist, which forces the dependency graph to stay accurate.


buyer beware, but buy smart


   
ReplyQuote
(@emilyw)
Reputable Member
Joined: 3 months ago
Posts: 188
 

Love the idea of treating prompts like internal libraries with unique IDs. But how do you handle versioning in practice? Say I need to update a prompt that ten services use - do you just roll out a new version and force all services to update their configs, or is there a way to manage backwards compatibility?



   
ReplyQuote
(@amandaf)
Reputable Member
Joined: 3 months ago
Posts: 455
 

Exactly, cost and risk are the right axes for sorting, not internal politics. But I've seen teams run into trouble when those metadata fields become static guesses. If a prompt's `estimated_volume` and `risk_tier` aren't validated against actual production logs, you get a false sense of security.

You need a process that compares that declared `cost_center` and `risk_tier` against the reality of which services are calling it and what data they're sending. Otherwise, a prompt tagged `pii_handling` gets called by a new, unvetted service and governance falls apart. The metadata block is the start, not the finish.


—AF


   
ReplyQuote
(@amandap)
Estimable Member
Joined: 3 months ago
Posts: 173
 

That makes sense, having services declare their dependencies. But what do you do when a prompt needs a change and you have to find all those services to update their configs? Is your script fast enough for that?



   
ReplyQuote
(@andrew8)
Reputable Member
Joined: 3 months ago
Posts: 365
 

Folders by department will fall apart. Organize by the data contract.

Our structure is `/{task-type}/{service-or-output}.yaml` (e.g., `/classification/support_ticket_v2.yaml`). Each file has a mandatory metadata block:

```yaml
id: classification.support_ticket
risk_tier: pii_handling
consumers:
- service_a
- workflow_b
```

We run a pre-commit hook that validates `id` uniqueness and parses the `consumers` list against our service registry. It blocks the commit if a referenced consumer doesn't exist.

Tags for model and department are secondary fields in the metadata. The primary key is the `id`.


Numbers don't lie.


   
ReplyQuote
(@hannahm)
Reputable Member
Joined: 3 months ago
Posts: 217
 

That's a really good point about the IDE plugin. The friction from waiting for a pre-commit hook is real, especially when you're trying to iterate quickly on wording.

But setting up a linter everyone uses feels like a different kind of challenge, at least for my team. We all use different editors. Is there a good way to enforce that kind of tooling without becoming the IDE police?


Just my two cents.


   
ReplyQuote
Page 3 / 7