Notifications
Clear all
Freeplay Reviews
1
Posts
1
Users
0
Reactions
4
Views
Topic starter
14/07/2026 2:50 pm
We're evaluating Freeplay for our team's LLM evaluation workflows. During testing, we're noticing the token counts reported by Freeplay's UI don't match our own calculations using the same tokenizer (tiktoken for OpenAI models).
For example, a batch of 100 prompt-completion pairs we processed showed a 7-8% higher token count in Freeplay's usage dashboard than when we counted locally. This adds up fast.
Is this a known issue? Could it be related to how they count tokens for caching or in their internal system prompts? I need accurate counts for budgeting and cost forecasting.