Everyone's raving about DeepSeek's low token prices. Price per million tokens isn't the metric that matters. It's price per *useful* output.
So I logged every API call for a month. Filtered out the "As an AI, I cannot..." and the endless disclaimers. Measured the actual, usable character count from JSON, code, or concise answers.
Results? Not great. The "useful" output was about 65% of the billed completion tokens on average. For creative tasks, worse. You're paying for a lot of boilerplate safety padding and waffle. That "cheaper" model starts looking a lot closer to the big vendors when you account for the noise.
Built it with a simple proxy to strip headers and track counts. Might open-source the tracker if there's interest. The real cost is in the verbosity and the pre-padded refusals.
—aB
—aB
Interesting approach. I've seen teams try to measure "signal to noise" ratios in LLM outputs, but usually with manual sampling. Automating it through a proxy is clever.
Did you track this per-endpoint or per-use-case? I'd expect big differences between something like a structured data extraction task versus open-ended brainstorming. The 65% average could hide wild swings.
If you open-source the tracker, I'd be curious how you handle edge cases - like when the "useful" output is technically there but buried in repetitive phrasing. That still adds cognitive overhead even if it's not a full disclaimer.
Per-endpoint tracking would be misleading, as the same chat/completions endpoint can serve wildly different functions. I grouped by inferred use case, tagging calls with metadata from the request body - things like the presence of a JSON schema, temperature settings, or keywords in the system prompt. The variance was indeed massive, from 90%+ efficiency in strict function-calling scenarios down to 40% for creative writing.
Your edge case about buried repetition is a good one. My current logic uses regex to strip known refusal patterns and disclaimers, but semantic repetition is harder. I'm experimenting with a simple Levenshtein distance check on consecutive sentences as a filter for "cognitive overhead" content. It's imperfect, but it catches the worst offenders.
Open-sourcing it would require cleaning up those heuristic rules, as they're currently tuned to our specific prompts. The core proxy that logs and parses the stream is the straightforward part.
Measure twice, cut once.