When evaluating the cost of using large language model APIs, the most fundamental unit of accounting is the token. However, providers universally split their pricing into two distinct categories: input (or prompt) tokens and output (or completion) tokens. This distinction is not merely semantic; it reflects underlying computational realities and directly impacts your cost structure, especially for workloads with asymmetric prompt-to-response ratios.
At a technical level, processing input tokens and generating output tokens involve different phases of the transformer architecture's workload. Processing the input prompt primarily involves the *encoder* or *context processing* phase, where the model attends to all tokens in the prompt simultaneously to build a contextual understanding. Generating output tokens is an *auto-regressive* process, where the model predicts the next token sequentially, with each new token being conditioned on all previous input and generated tokens. This sequential nature makes generation more computationally intensive per token, which is a key reason for the typical pricing disparity.
Consider a practical example using pseudo-code for a hypothetical API call:
```python
response = llm.complete(
prompt="Explain the theory of relativity in simple terms.", # These words are input tokens.
max_tokens=500 # The response will consist of up to 500 output tokens.
)
```
If the prompt is 10 tokens and the generated response is 500 tokens, you are billed separately for each count. With a common pricing model of, say, $0.50 per 1M input tokens and $1.50 per 1M output tokens, the cost calculation would be:
* Input cost: (10 / 1,000,000) * $0.50 = $0.000005
* Output cost: (500 / 1,000,000) * $1.50 = $0.00075
The total cost is dominated by the output tokens, despite the prompt being essential.
This pricing model has significant implications for architectural decisions:
* **Retrieval-Augmented Generation (RAG):** Injecting large context windows (e.g., entire documents) into the prompt drastically increases input token count. A provider with relatively cheaper input tokens may be more economical for such tasks.
* **Summarization vs. Expansion:** Summarizing a long document (many inputs, few outputs) has a very different cost profile than generating a long story from a short prompt (few inputs, many outputs).
* **Chat Completion Sessions:** In a multi-turn chat, the entire conversation history is typically re-submitted with each new user message, causing cumulative input token growth. Some providers offer mechanisms like "caching" to mitigate this.
Therefore, when comparing providers like OpenAI, Anthropic, or Google Vertex AI, you must examine both rates. A provider with a lower output token rate might be superior for generative tasks, while one with a lower input token rate could be better for analysis of large texts. Always model your expected usage pattern—average prompt length and expected response length—to project costs accurately. Treating them as separate line items is as crucial as understanding read vs. write costs in a managed database service.
SQL is not dead.