In our ongoing work to optimize production LLM pipelines, we've found that comparing models purely on token cost or even simple accuracy metrics is insufficient. It fails to capture the economic reality of deploying these systems at scale: you are paying for an *acceptable* output, not just any output. A cheaper model that requires three retries or significant post-processing to produce a usable result may be far more expensive than a pricier, more reliable one.
To address this, our team has formalized a metric we call **Cost per Acceptable Output (CPAO)**. It is designed to be calculated over a statistically significant evaluation run and incorporates both the raw API costs and the model's performance against our specific quality gates.
The CPAO is calculated as follows:
```
CPAₒ = (Σ (Callsᵢ * CostPerCallᵢ)) / Σ AcceptableOutputsᵢ
```
Where:
* `Callsᵢ` is the number of API calls made to a given model for a specific task.
* `CostPerCallᵢ` is the total cost for that call (input + output tokens, often with a premium for longer context).
* `AcceptableOutputsᵢ` is a binary (1 or 0) determined by our evaluation framework for that specific task output.
The critical component is, of course, the definition of "acceptable." This is not a generic benchmark score but a task-specific rubric aligned with business logic. For example:
* **Customer Support Intent Classification:** Acceptable = correctly identifies primary intent AND extracts all required entities with >95% confidence.
* **Code Generation:** Acceptable = compiles without errors, passes all pre-defined unit tests, and adheres to style guidelines.
* **Summarization:** Acceptable = retains all key factual claims from source, demonstrates high ROUGE-L score against human reference, *and* no hallucination of new facts.
We run a batch of *N* representative prompts through each model API, apply our evaluators (a mix of automated checks, smaller LLM-as-a-judge calls, and sampled human review), and tally the acceptable outputs. The total cost divided by that tally yields the CPAO.
A practical example from a recent comparison for a document Q&A task:
| Model API | Cost per 1k Output Tokens | Avg. Output Tokens per Call | % Acceptable (Our Rubric) | **CPAₒ** |
| :--- | :--- | :--- | :--- | :--- |
| Provider A (top-tier) | $0.030 | 450 | 94% | **$0.014** |
| Provider B (mid-tier) | $0.015 | 520 | 71% | $0.016 |
| Provider C (budget) | $0.008 | 620 | 42% | $0.032 |
Despite Provider B having half the per-token cost of Provider A, its lower acceptability rate made it marginally more expensive per *good* answer. Provider C, while cheapest on paper, was economically unviable for this task due to its high failure rate.
This metric forces a shift from isolated cost and quality metrics to a unified efficiency view. It directly answers the operational question: "What am I actually paying for each unit of work this pipeline completes successfully?" We've integrated CPAO tracking into our MLOps dashboards, where it is calculated per model, per task, and even per client, allowing for granular optimization.
I'm interested to hear how others are approaching this problem. Are you using similar composite metrics? What methodologies are you employing to define and automate the "acceptability" judgment at scale, particularly for subjective tasks?
—BJ
—BJ