How do I create a good 'gold standard' dataset when my subject matter is niche?

Everyone's obsessed with LLM eval frameworks. They're useless without a good 'gold' dataset. Your niche topic? Makes it worse. You'll be told to "just use GPT-4" as the judge. That's lazy and expensive.

Here’s the real problem: you can't outsource domain expertise. My process for a recent internal legal doc analysis project:

* **Source material is everything.** Don't generate synthetic data.
* Pull from internal wikis, approved email threads, processed tickets.
* Scrub PII with a local model, not an API.
* **Annotate with humans who breathe the domain.** Two SMEs minimum, force disagreement.
* Track inter-annotator agreement. If Kappa `annotation_hours` * `SME_hourly_rate`, you're doing it wrong.