The premise of this thread suggests a focus on user experience and workflow, but from a benchmarking perspective, this practice is a foundational step in establishing a controlled, reproducible evaluation environment. I do not maintain a simple text file; I maintain a structured repository of prompt templates, variables, and corresponding evaluation criteria. The goal is not merely to "have good prompts," but to have a test suite that allows for the comparative analysis of model performance across versions of HuggingChat, or against other hosted LLMs, under identical conditions.
My primary "text file" is a YAML configuration that defines prompt templates, systematic variations, and the expected evaluation metrics. For example, when benchmarking reasoning capabilities, I will have a base template and then iterate over parameters like temperature and top_p to measure consistency versus creativity. A simplified snippet of such a structure:
```yaml
benchmark_suite: "logical_deduction_v1"
model_target: "HuggingChat (Meta-Llama-3-70B-Instruct-Tuned)"
prompt_templates:
- name: "syllogism_5step"
template: |
You are a precise logic engine. Deduce the conclusion from the following premises step by step.
Premise 1: {{premise1}}
Premise 2: {{premise2}}
Premise 3: {{premise3}}
...
Show each deductive step. Conclude with "Therefore: [clear conclusion]".
variables:
premise_sets: [list_of_premise_tuples]
generation_params:
- {temperature: 0.1, max_new_tokens: 500}
- {temperature: 0.7, max_new_tokens: 500}
evaluation_metrics:
- "formal_validity"
- "step_completeness"
- "hallucination_check"
```
The prompts themselves are engineered not for a single "good" response, but to elicit responses that can be programmatically or semi-automatically scored. I am interested in prompts that:
* **Isolate a specific capability** (e.g., instruction following, code generation, factual recall under constraints).
* **Allow for the injection of variables** to test generalization (e.g., different coding problem statements, different factual questions from a known corpus).
* **Enforce a structured output format** (JSON, specific section headers) to simplify automated parsing and scoring.
For instance, a prompt I use to benchmark the retrieval-augmented generation (RAG) simulation performance is:
```
You will answer the question based solely on the provided context. Your response must contain two sections:
1. "Answer:" [a direct, concise answer to the question].
2. "Supporting Sentences:" [a bulleted list of *exact* quotes from the context that were used to formulate the answer].
If the answer cannot be determined from the context, state "Unanswerable from context."
Context: {{chunk_of_text}}
Question: {{question}}
```
This allows me to compute precision/recall on the "Supporting Sentences" against a known ground truth.
My question to the community is not simply "share your prompts," but rather: **What measurable characteristics are you optimizing for with your saved prompts, and how do you quantify the performance delta between using a saved "effective" prompt versus a naive one?** Are you tracking latency, token usage (critical for cost-per-query), correctness rate, or user rating distributions? Without this quantitative lens, we are merely trading anecdotes.
numbers don't lie
numbers don't lie