Just ran a benchmark on DeepSeek Chat's system prompt capacity. The context window is one thing, but the real test is how much of that you can use to lock the model into a specific operational persona before it starts ignoring instructions or getting "creative."
I pushed a 1500+ token system prompt into it, defining a strict JSON-only responder for a mock CRM API. The prompt included:
* Output format specification (strict JSON, specific keys)
* Rules for data validation (required fields, type checking)
* Error handling structure
* A prohibition on any explanatory text
The result? It adhered to the format for 12 consecutive queries before a slight deviation on the 13th (added a trailing comma outside the JSON). Not perfect, but significantly more constrained than other models I've tested with similar "jailbreak" attempts.
Key finding: You can effectively turn it into a single-function API stub. Useful for prototyping integration logic without spinning up a local server.
Example of the prompt structure I used:
```json
{
"role": "system",
"content": "You are a CRM API endpoint. You accept user input as if it were a POST request. You MUST output ONLY valid JSON with the following structure: {"data": {...}, "errors": []}. Validate the input for 'email' and 'name' fields. If 'email' is missing or invalid, add an error message to the errors array. Never add commentary."
}
```
The practical implication for CRM work:
* You could simulate a third-party service's API during migration planning.
* Generate realistic test payloads based on the same constraints.
* Stress-test your own API specs by making the model act as the client.
Has anyone else pushed the system prompt to enforce a rigid output schema? What's the breakpoint where it starts to ignore significant portions? I'm looking for reproducible tests, not anecdotes.
Show me the query.
That's a really neat use case! I've done similar tests but with more infrastructure-focused personas, like a Terraform plan analyzer that only outputs in HCL or a Dockerfile linter with strict rule sets.
The 12-query adherence is impressive. I've noticed the "creep" usually starts with minor formatting slips (like your trailing comma) before any major rule breaks. Makes me wonder if there's a kind of "instruction fatigue" setting in after a certain number of exchanges.
Have you tried mixing in a periodic "reset" instruction in the user prompt? Sometimes I'll add a quick "Remember: JSON only, no commentary" on every 5th query or so in my scripts, which seems to extend the strict adherence window. Not ideal for a pure API simulation, but it helps for longer sessions.
Keep deploying!