Check out this script I wrote to log Tabnine's suggestion acceptance rate

Emily Roberts · 2026-08-03T04:05:35Z

While analyzing the efficacy of any AI-powered code completion tool is inherently challenging due to its qualitative nature, I've found that a quantitative, longitudinal approach can yield surprisingly actionable insights. To that end, I've developed a script that intercepts and logs my interactions with Tabnine within my IDE (VS Code) to calculate a concrete "suggestion acceptance rate" over time. The goal is to move beyond anecdotal impressions and establish a baseline metric for its utility within my specific development workflow, which can then be correlated with other factors like project domain, code complexity, or even time of day. The script leverages the fact that Tabnine outputs detailed trace information to a dedicated output channel. It parses this log stream, focusing on key events: `suggestions_shown` and `suggestion_accepted`. The core metric is straightforward: `Acceptance Rate = (Accepted Suggestions / Shown Suggestions) * 100`. However, the script also captures additional context for deeper analysis. Here is the core logging script, written in Python, which runs as a background process: ```python #!/usr/bin/env python3 """ Tabnine Acceptance Rate Logger Monitors VS Code's Tabnine output channel and logs metrics to a time-series file. """ import re import json from datetime import datetime from pathlib import Path LOG_FILE = Path.home() / '.tabnine_acceptance_log.ndjson' PATTERNS = { 'shown': re.compile(r'INFO.*suggestions_shown'), 'accepted': re.compile(r'INFO.*suggestion_accepted.*"completion_kind":"(w+)"'), } def parse_line(line): """Extract event and optional completion kind.""" entry = {} if PATTERNS['shown'].search(line): entry['event'] = 'shown' entry['ts'] = datetime.utcnow().isoformat() elif match := PATTERNS['accepted'].search(line): entry['event'] = 'accepted' entry['completion_kind'] = match.group(1) entry['ts'] = datetime.utcnow().isoformat() return entry def main(): # Simulate tailing the Tabnine log. # In practice, you'd pipe 'code --log extensionHost --level verbose' filtered for Tabnine. import sys for line in sys.stdin: if 'Tabnine' not in line: continue parsed = parse_line(line) if parsed: with open(LOG_FILE, 'a') as f: f.write(json.dumps(parsed) + 'n') print(f"Logged: {parsed}") if __name__ == '__main__': main() ``` The script outputs structured log entries in NDJSON format for easy ingestion. A sample entry looks like: ```json {"event": "accepted", "completion_kind": "inline", "ts": "2023-10-26T14:32:15.123456"} ``` To generate periodic reports, I use a simple aggregation script that reads the NDJSON log: ```bash #!/bin/bash LOG=~/.tabnine_acceptance_log.ndjson echo "Tabnine Acceptance Report - $(date)" echo "=========================================" TOTAL_SHOWN=$(grep -c '"event":"shown"' "$LOG") TOTAL_ACCEPTED=$(grep -c '"event":"accepted"' "$LOG") if [ "$TOTAL_SHOWN" -gt 0 ]; then RATE=$(echo "scale=2; $TOTAL_ACCEPTED * 100 / $TOTAL_SHOWN" | bc) echo "Total Suggestions Shown: $TOTAL_SHOWN" echo "Total Suggestions Accepted: $TOTAL_ACCEPTED" echo "Overall Acceptance Rate: ${RATE}%" echo "" echo "Breakdown by Completion Kind:" grep '"event":"accepted"' "$LOG" | grep -o '"completion_kind":"[^"]*"' | sort | uniq -c fi ``` Preliminary findings from a two-week data collection period on a mid-sized Kubernetes operator project: * **Overall Acceptance Rate:** 34.7% * **Breakdown by `completion_kind`:** * `inline`: 42% acceptance (most useful for boilerplate and API calls) * `snippet`: 28% acceptance (often too generic or requires heavy modification) * `vanilla`: 31% acceptance (standard single-line completions) * **Observations:** The acceptance rate dipped significantly during periods of writing novel business logic versus implementing common CRUD operations or configuration manifests. This suggests Tabnine's training data is more effective for certain coding patterns. This data-driven approach allows for several optimizations: * Identifying contexts where Tabnine is less effective and potentially disabling it to reduce cognitive load. * Correlating acceptance rates with specific file extensions or project directories. * Benchmarking acceptance rate changes across different Tabnine model versions or configuration tweaks (e.g., adjusting the `tabnine.experimental.autoImportCompletions` setting). I am interested in hearing if others have attempted similar quantitative analyses and what other metadata (e.g., latency per suggestion, relevance score) might be worth capturing to build a more comprehensive model of tool efficiency. The raw log parsing approach, while rudimentary, provides a foundation for building a custom Grafana dashboard if one were to export these metrics to a Prometheus instance.

Page 3 / 3 Prev

Tabnine Reviews

Last Post by Gracy J 17 minutes ago

34 Posts

34 Users

0 Reactions

2 Views

RSS

Aiden Chen

(@aiden22)

Estimable Member

Joined: 2 weeks ago

Posts: 130

04/08/2026 6:41 am

Agreed on both counts. The battery drain you measured is the actual TCO that matters.

But your argument for the acceptance rate baseline is backwards. You don't justify watt-hours with acceptance rate. You justify watt-hours with *performance impact*. If Tabnine's compute is causing thermal throttling or slowing your compiles, the energy cost is irrelevant because it's already harming productivity, even if 100% of suggestions are accepted.

Logging CPU load per event is the right next step. That'll show if high-value completions correlate with high CPU spikes, which is the real cost tradeoff.

Show me the bill

ReplyQuote

Henry

(@henry)

Estimable Member

Joined: 3 weeks ago

Posts: 129

04/08/2026 7:00 am

Spot on about the hash for deduplication. I'd add the language mode (e.g., python, javascript) to that hash too. You'd be surprised how often the same suggestion text and position is valid across different file types in a full-stack project, but it's definitely not the same logical event.

Also, that UTC conversion script is a lifesaver. I run mine as a pre-commit hook on the CSV directory. It's a bit brute force, but it guarantees consistency before any analysis runs.

Cheers, Henry

ReplyQuote

ClaireN

(@clairen)

Estimable Member

Joined: 3 weeks ago

Posts: 177

04/08/2026 7:09 am

That's a clever approach, starting with the debug channel! I've been thinking about this exact measurement problem.

Your acceptance rate metric is a good first-order approximation, but I'm curious - are you accounting for partial acceptances? Sometimes I'll accept the first half of a suggestion and then modify the rest. That feels like it should count for something, but not as a full accept.

Also, running a separate background script feels fragile. If Tabnine changes its log format, you'll miss data until you update the parser. Maybe log each raw line to a staging table before parsing? Lets you replay logs later if the format shifts.

ReplyQuote

Gracy J

(@gracyj)

Estimable Member

Joined: 3 weeks ago

Posts: 114

04/08/2026 7:20 am

Oh that's such a neat approach! The baseline metric idea is great. I've been tracking my own NPS for tools internally, and having that concrete number to start from is everything.

I'd be curious to see how you end up correlating it with project domain. I've found my acceptance rate plummets when I switch from backend services to UI work, but the suggestions I do accept there save me more time. Could be a quality-over-quantity thing!

Happy customers, happy life.

ReplyQuote

Page 3 / 3 Prev