I've run into a classic but frustrating scenario with Playground AI's fine-tuning API, and I need to see if anyone else has done the systematic legwork to figure this out. The promise is straightforward: take a base model (in my case, `llama-3.2-3b-instruct`), fine-tune it on a high-quality, domain-specific dataset, and get a model that excels at that domain. The reality I'm measuring is a model that underperforms its base counterpart on both general capability and, more damningly, on its *specific* trained task.
Here's my setup and the synthetic benchmarks I ran:
**Base Model:** `llama-3.2-3b-instruct` (Playground AI's hosted version)
**Fine-tuned Model:** Fine-tuned version of the above on ~10,000 examples of structured technical Q&A (format: question -> answer with code snippet and explanation).
**Training Config:**
```
{
"base_model": "llama-3.2-3b-instruct",
"training_epochs": 3,
"batch_size": 4,
"learning_rate": 2e-5,
"lora_r": 16
}
```
The training loss decreased cleanly, and Playground's dashboard reported no obvious issues.
**Evaluation Method:**
I created a held-out test set of 500 examples from my domain and 200 general MMLU-style questions. I used a consistent inference configuration for both models:
- temperature: 0.0
- max_tokens: 1024
- top_p: 1.0
I then scored outputs using:
1. **Exact match** for code snippets.
2. **BERTScore F1** for answer semantic similarity.
3. **Latency** (p50, p95) and **cost per 1k output tokens** (tracked via Playground's usage dashboard).
**Results (Averaged):**
| Metric | Base Model | Fine-tuned Model | Delta |
| :--- | :--- | :--- | :--- |
| **Domain Code Exact Match** | 72.4% | **65.1%** | **-7.3%** |
| **Domain BERTScore F1** | 0.891 | 0.843 | -0.048 |
| **General MMLU Accuracy** | 64.5% | 58.2% | -6.3% |
| **P50 Latency (ms)** | 142 | 155 | +13 |
| **Cost per 1k Tokens** | $0.05 | $0.07 | +40% |
The fine-tuned model is objectively worse across the board—it's less accurate, slightly slower, and significantly more expensive to run. This suggests catastrophic forgetting or an overfitting regime where the model has memorized the training data patterns but lost its general reasoning and instruction-following ability, which is critical for my task.
My hypotheses, but I lack the internal visibility to confirm:
* **Overfitting on sequence structure:** The fine-tuning data had a rigid template. The model may have learned to replicate the template even when it's inappropriate for the specific query.
* **Learning rate / epochs misconfiguration:** Playground's defaults might be too aggressive for a 3B model on a 10k-example dataset, causing rapid forgetting.
* **Base model weight contamination:** If the fine-tuning is a LoRA adapter, is there an inference-time bug where the base weights aren't being loaded correctly? The latency and cost increase suggest *something* different is happening.
Has anyone conducted a similar A/B test with quantifiable outcomes on Playground? I'm particularly interested in:
* Your evaluation methodology.
* Any successful configuration changes (lower LR, more epochs with dropout, etc.).
* Whether you found a "sweet spot" for dataset size relative to model size on their platform.
* If you've seen this pattern with other base models (like `mixtral-8x7b` or `qwen-2.5-32b`).
Without reproducible, measured results, we're just trusting the platform's black box. I'd like to change that.
Show me the benchmarks