Our development team has been scaling up its usage of various LLM APIs (primarily OpenAI GPT-4 and Claude 3 Opus) for a range of internal tooling, and we observed a significant variance in the operational costs of similar tasks. To foster better practices and provide visibility, I spearheaded an initiative to create an internal "Prompt Efficiency Leaderboard." The core metric is a simple but effective cost-per-task unit, calculated by dividing the total cost of API calls for a given task by the number of successful, validated task completions.
The implementation leverages PromptLayer as the central observability layer. Its ability to log every call, along with tokens, cost, and metadata, made aggregating this data straightforward. We built a simple internal dashboard that queries PromptLayer's data via their REST API, grouping results by developer tag and task type.
The key schema for our scoring is as follows:
```python
# Pseudo-code for the core efficiency metric calculation
def calculate_developer_score(developer_tag, start_date, end_date):
# Fetch all logs for the developer via PromptLayer API
logs = promptlayer.get_logs(
tags=[developer_tag],
start_date=start_date,
end_date=end_date
)
# Group logs by 'task_id' (a custom tag we enforce)
tasks = group_logs_by_task(logs)
total_cost = 0
successful_completions = 0
for task_id, calls in tasks.items():
task_cost = sum(call.cost for call in calls)
total_cost += task_cost
# Task is considered successful if the final call has a validation flag
if calls[-1].metadata.get('validation_passed'):
successful_completions += 1
# Avoid division by zero
if successful_completions == 0:
return float('inf')
cost_per_successful_task = total_cost / successful_completions
return cost_per_successful_task
```
We enforce tagging by requiring developers to add their unique identifier and a task classification (e.g., `code_review`, `sql_gen`, `summarization`) in the PromptLayer `tags` field for any production-bound call. The dashboard then displays:
* **Primary Leaderboard:** Ranks developers by aggregate cost-per-successful-task across all task types (lower is better).
* **Per-Task Breakdown:** Shows efficiency rankings within specific task categories, as some are inherently more expensive.
* **Key Metrics Per Developer:**
* Total spend for the period
* Average tokens per call (input and output broken out)
* Number of failed/completion tasks (via our validation metadata)
Initial findings after two weeks have been revealing. The variance in cost-per-task for similar outcomes exceeded 300% across the team. The leaderboard has already driven constructive competition and knowledge sharing, with top performers conducting "code reviews" on prompt chains. Notably, we've observed a rapid standardization on more efficient system prompt patterns and a reduction in unnecessary verbose output structures.
I am considering refining the metric to incorporate a latency-weighting factor, as some solutions trade cost for excessive time-to-completion. I'm curious if others in the community have implemented similar internal benchmarking systems and what additional dimensions (reliability scores, retry rates) you found valuable to track.
-- bb42
-- bb42