Hello everyone. I've noticed a lot of discussions about Codeium's utility in individual workflows, but as a team lead, my primary concern is understanding adoption and impact at the team level. While the Codeium dashboard provides high-level metrics, I found myself wanting a more granular, persistent, and team-oriented view. So, I spent the last weekend building an internal dashboard to track our engineering team's Codeium usage, and I thought I'd share the approach and some early insights.
The core idea was simple: periodically fetch data from the Codeium API, store it in a time-series database, and visualize trends. The goals were to track:
* **Overall Adoption:** How many of our developers are actively using Codeium day-to-day?
* **Feature Breakdown:** Are we leaning more on completions, chat, or search? This helps tailor internal training.
* **Language & Project Insights:** Which codebases or languages see the highest engagement? This can highlight where Codeium is most effective or, conversely, where it might be struggling.
* **Trends Over Time:** Are weekly accepted completions going up? Did a recent workshop cause a spike in chat usage?
I set up a simple Python service that runs daily via a scheduled job. It uses the Codeium API endpoints for user and event metrics (with appropriate, anonymized access tokens). The data is cleaned and pushed into a small PostgreSQL instance with a `TimescaleDB` extension for efficient time-series queries. The frontend is a basic `Grafana` dashboard, which is perfect for this kind of operational data.
Here's a simplified version of the core fetching logic:
```python
import requests
import pandas as pd
from datetime import datetime, timedelta
def fetch_team_metrics(api_key, team_id, start_date):
headers = {'Authorization': f'Bearer {api_key}'}
params = {
'team_id': team_id,
'start_time': start_date.isoformat(),
'granularity': 'DAY'
}
# Fetch completion events
response = requests.get('https://api.codeium.com/v1/teams/metrics/events', headers=headers, params=params)
events_data = response.json().get('events', [])
# Process and transform data
df = pd.DataFrame(events_data)
# ... aggregation logic ...
return df
```
Some early, and quite positive, observations:
1. **Adoption is climbing steadily,** but not uniformly. Two developers who were initially skeptical showed a significant uptick in accepted completions after we shared some project-specific prompt tips.
2. **Chat usage is strongly correlated with onboarding new services or delving into legacy code.** We see clear spikes when someone is assigned a ticket in an unfamiliar part of the monorepo.
3. **The "lines accepted" metric is useful, but context is king.** A high count in a boilerplate-heavy configuration language (like YAML) is different from a high count in our core business logic services. We're starting to weight them differently in our analysis.
This isn't about surveillance—it's about understanding and enabling. The data already helped us identify a need for a brief internal session on crafting better prompts for Codeium Chat, which we held last Friday. The next step is to correlate this data with PR cycle time and maybe even sentiment from retrospective notes to get a feel for impact on developer satisfaction.
I'm curious if anyone else has taken a similar approach to measuring AI coding tool integration at the team or org level. What metrics are you finding most meaningful? Have you hit any snags with the API or data interpretation?
—Felix
This is a fantastic initiative. As someone who's often on the procurement and vendor management side, I immediately think about how valuable this granular data is for contract renewals and value justification. The Codeium-provided dashboard is good, but having your own persistent dataset lets you answer specific questions from finance or management.
Your point about tracking which *codebases or languages see the highest engagement* is key. It could reveal if adoption is siloed, or if there are specific projects where the tool is failing to gain traction. That data is crucial for deciding whether to push for broader training, or to have a targeted conversation with the vendor about support for a particular tech stack.
Are you planning to correlate this usage data with any other team metrics, like pull request throughput or cycle time, to get at that "impact" question? I'm curious if you've thought about a framework for that.
buyer beware, but buy smart
Hey, I really appreciate you sharing this. Tracking adoption at a team level is such a common pain point that goes unaddressed by most vendor dashboards, and your approach of pulling from the API into your own time-series storage is the right way to get a persistent, historical view. I've seen teams try to rely on screenshots or monthly reports, and it always falls apart.
One thing your breakdown made me consider is the "why" behind the trends you'll see. For example, if accepted completions go up, is it because the tool is getting better, or because a new engineer joined who uses it heavily? Having that historical baseline will let you ask those questions. I'm also curious, when you start seeing the data, will it change how you roll out training or communicate about the tool internally? Like, if you see a drop in chat usage, would that prompt a quick internal reminder about the feature?
Excited to see what you learn from this.
Let's keep it real.
Good approach pulling data into your own time-series store. The Codeium dashboard resets on their schedule, not yours.
Track accepted vs. shown completions. A high shown count with low acceptance means the suggestions aren't useful, which directly impacts productivity calculations.
Are you logging costs? You need to correlate usage spikes with your seat-based billing. A 50% spike in accepted completions is great, but not if it just means your bill doubles next renewal.
Show me the bill
You've hit on the key procurement point. Having an independent, historical dataset is the only way to negotiate effectively. A vendor's dashboard will always show their best-performing timeframe by default.
>Are you planning to correlate this usage data with any other team metrics
That's the logical next step for measuring impact, but it's tricky. Correlating usage with PR throughput or cycle time requires a controlled baseline to be meaningful. A spike in "accepted completions" might correlate with a drop in cycle time, but did the tool cause it, or did the team just finish a bunch of easy chores? You'd need to segment by task complexity, which most teams don't log.
A simpler, more defensible correlation for finance might be cost-per-accepted-completion over time. If that metric drops while usage rises, you've got a strong efficiency argument for renewal.
BenchMark
The language and project breakdown is a smart focus area, but I've found you need to normalize that data against commit volume to make it truly interpretable. A spike in Python completions might just mean that's the only language with active development that week.
What's your plan for data retention? With time-series data, you can quickly hit scale issues if you're storing granular events per developer. I'd recommend rolling up daily aggregates after a month or so, unless you specifically need minute-level granularity for anomaly detection.
Extract, transform, trust
Good point on normalization. But commits are also a vanity metric. You need to correlate against *meaningful* work, like story points or resolved tickets, otherwise you're just normalizing noise with more noise.
Daily rollups are fine for cost tracking, but you lose the ability to spot context. If a dev goes from 2 to otw accepted completions an hour, you need that granularity to see if it's a real change or just them debugging one file all afternoon.
If it's not a retention curve, I don't care.
That's a really solid use case for pulling from the API directly. The project and language breakdown is what I'd be most interested in too. Are you scoping the API calls by repository, or are you tagging the data post-fetch based on the developer's current project? I've had mixed results with that mapping.
Also, for tracking daily active users, are you defining "active" as any event, or only meaningful ones like accepted completions? A dev could have the extension open all day but not use it.
Data is the new oil - but it's usually crude.
Thanks for sharing your project, and for outlining those specific goals. That kind of clarity is what helps turn data into something actionable for a team.
The next hurdle for a lot of teams is ensuring this kind of analysis is done neutrally. Since you're building this internally, you're in a good spot to avoid confirmation bias. The key question becomes: are you tracking metrics that could show the tool *isn't* providing value in certain areas? For example, a persistently low acceptance rate for completions in a specific project might indicate a need for a different tool or approach there, not just more training.
Setting up the pipeline is the first step. Deciding how you'll act on negative or neutral trends, and communicating that to your team transparently, is where the real work begins.
Keep it constructive.
Cost-per-accepted-completion is the metric for procurement. That's what I track.
But it's a lagging indicator. You need leading signals too. Track the acceptance rate per language or per major project directory. A sustained drop there means the tool's suggestions are becoming less relevant, which will drive your cost-per-completion up long before the bill comes due.
Data over opinions
You're right that commits are noisy, but story points are fiction. I've tried correlating with Jira tickets, and the variance in what counts as a 'story point' between teams, even sprints, makes the data useless for anything but the broadest trends.
The granularity argument is key, though. If you roll up to daily, you can't isolate a refactoring session from a feature-building session. I keep raw events for 90 days for this exact reason. The storage cost is trivial compared to losing the signal. After that, I roll up to daily by developer and project for the long-term cost-per-completion trend.