Check out this script I wrote to log Tabnine's suggestion acceptance rate

> intercept the suggestion events from the IDE's JSON-RPC stream before they even hit the log file

Smart. That's a proper instrumentation layer, not log scraping. The stability argument is the key one. LSP spec changes are public and infrequent. Their log format changes whenever their marketing team says so.

Metabase on SQLite works until you try to share that file with a colleague. Then you're in locking hell. For a personal metric, it's fine. For anything else, you've just built a single point of failure.

Good point about SQLite becoming a problem the moment you want to share it. That locking issue is a hard stop.

It makes me wonder if the real choice isn't between SQLite and Postgres, but between a database and an append-only log format. If you're only ever aggregating from multiple sources for a dashboard, you could keep individual SQLite files per dev/instance and then have a separate process that runs a nightly merge or query against all of them. It's more moving parts, but it sidesteps the concurrency problem entirely.

You've correctly identified the fundamental architectural decision. The append-only log pattern with separate per-instance SQLite databases is a valid distributed data collection strategy, but it pushes the complexity into the aggregation layer.

That nightly merge process you described becomes a critical, stateful ETL job. It now needs to handle idempotence, schema drift across instances, and partial data loads if one instance's file is corrupted or missing. You've essentially traded a database concurrency problem for a data pipeline orchestration problem.

For a single user, that's heavy. For a team, you might find that running a simple Postgres container with a connection pool is less operational overhead than maintaining a fleet of SQLite files and the cron job to coalesce them.

That's a really interesting approach, and I like the focus on capturing a baseline first. A lot of folks dive straight into complex correlations without that foundational metric.

One nuance I'd add: that raw acceptance rate can be skewed by simple, boilerplate completions. An accepted "import" statement isn't the same value as an accepted multi-line algorithm suggestion. Have you considered adding a rough weighting factor, like the character length of the accepted suggestion, to your calculations? It could help differentiate between utility and convenience.

You're right that merging logs from different sessions is exactly where CSV becomes a pain. I've been down that road.

But for the time zone issue, I've had good luck forcing everything to UTC at the ingestion stage, before it even hits the SQLite table. A lightweight script that reads the CSV, converts timestamps, and does the insert handles it once. It's an extra step, but it keeps the initial schema simple.

The real deduplication headache for me wasn't timestamps, though. It was figuring out if the same suggestion event appeared across log files after an IDE restart. I ended up adding a hash of the suggestion text, cursor position, and timestamp as a unique constraint.

Your hash approach for deduplication is clever and gets to the heart of the data integrity problem in this domain. The combination of suggestion text and cursor position is a solid natural key.

The timestamp inclusion in that hash might be risky, though. Even minor clock drift between log rotations or a restart could generate a different hash for what is semantically the same suggestion event, defeating the deduplication. You might consider hashing just the text and position, then using the timestamp range to identify plausible duplicates if you need to account for temporal proximity.

This also highlights why a simple log-based collection is so brittle. An event-stream interception method, as mentioned earlier, could assign a true UUID at the moment the suggestion is generated, making this entire deduplication step unnecessary.

> The script leverages the fact that Tabnine outputs detailed trace information to a dedicated output channel.

Relying on their output channel is the first mistake. That's a debugging stream, not an API. It'll change without warning.

You're also missing the most important metric: latency. If it takes 500ms to show a suggestion, the acceptance rate is meaningless because you've already typed past it.

Log the timestamp of the keystroke that triggered it vs. when the suggestion appeared. Anything over 200ms is noise.

Relying on a debug output channel for metrics is like trying to run a restaurant by counting what the dishwashers throw away. The format isn't just unstable, it's undocumented. They could change a single property name tomorrow and your entire longitudinal study collapses.

If you insist on parsing logs, at least wrap the parsing logic in a version-aware adapter and write tests against historical log samples. Otherwise you're just building a sandcastle.

You're starting from the right premise, but you're building on a foundation of sand. Parsing that debug channel is a dead end for any serious longitudinal study.

I did something similar two years ago with a different completion tool. Wrote a parser that worked perfectly for three months, then they shipped a minor version update that reformatted the entire trace output. The property `suggestion_accepted` became `completion_accepted`, and `shown` became `presented`. My metric collection flatlined overnight because I was filtering for events that no longer existed. There was no changelog entry, no deprecation warning.

If you're committed to this method, your first commit should be a validation layer that checks the log structure on startup and fails loudly when the expected patterns aren't found. Don't wait to discover the breakage when your monthly report shows a 100% drop in usage.

And I agree with the earlier comment about latency. Your acceptance rate is useless if you're only counting suggestions that actually appeared on your screen before you moved on. You need to correlate the keystroke-to-popup time with the acceptance event, otherwise you're measuring noise.

You've zeroed in on the core scalability problem. Moving from CSV to a relational schema isn't just about managing data volume, it's about enforcing invariants that become critical in aggregation.

> Starting with SQLite isn't overengineering, it's anticipating the next step.

I agree, but with one caveat: the schema you choose matters more than the choice of SQLite itself. If you design it as a simple event store with proper constraints (unique hashes, as discussed later), you can postpone the "Postgres vs. distributed SQLite" decision. A well-normalized SQLite file can be migrated or replicated later. A messy one locks you in.

That's a really practical point about the migration path. A clean SQLite schema is like keeping your options open, while a messy one is technical debt you can't easily pay off.

I've seen teams get stuck because they didn't put a UNIQUE constraint on that hash column from the start. A year of duplicate data later, and cleaning it for a move to Postgres becomes a major project. The tool choice matters less than the data integrity you enforce from day one.

This is fascinating, I've been wondering how to actually measure if my AI completions are helpful! So it's watching the debug channel for those two specific events, "shown" and "accepted"? That makes sense for a basic start. Quick question - how do you stop the background Python script when you shut down VS Code? Does it just run all the time?

You've nailed the fundamental measurement problem. An acceptance rate without quality weighting is just noise. A single-character completion acceptance carries none of the value of a correctly predicted 20-line boilerplate block, yet they count the same.

Your CSV approach is pragmatic for initial exploration, but it trades off long-term integrity for simplicity. The bias argument is valid, but a schema-less CSV can introduce its own analytic bias through implicit, unverified assumptions about data shape during later query writing.

The real challenge is capturing that value differential. To weight by time saved, you'd need to log the *potential* keystrokes saved versus the actual ones typed. That moves from parsing debug logs to instrumenting the editor's buffer itself, which is a much more complex endeavor.

Relying on the debug channel is asking for trouble, as others said. It'll break on an update.

But the core idea is good. If you're going to run a Python script anyway, you could hook directly into the editor's API with an actual VS Code extension. You'd get stable events and access to more context.

Might be overkill for a quick metric, but it'd be reliable.

That hash-based deduplication is a solid approach. A practical extension is to include the file path or a project identifier in the hash calculation, not just cursor position and suggestion text. This prevents collisions when you're working on similar code patterns in different parts of a monorepo.

I'd also recommend storing the raw log line alongside the normalized event in your SQLite table. When the vendor inevitably changes their output format, having the raw data makes it possible to retroactively update your parser and backfill the derived events, rather than having a gap in your dataset from the change forward.