hi everyone! 😅 i've been using copilot for a few months now and i love it, but i was curious... sometimes it feels like it suggests the same *type* of solutions over and over. i wanted to see if i could actually measure that.
so i made a little script that logs every suggestion copilot makes into a spreadsheet. it captures the prompt i typed and the full suggestion. my plan is to look for patterns, like if it always uses certain libraries or approaches. maybe it's biased towards certain coding styles? i'm still new to this, so i'm not sure what i'll find yet. has anyone else tried something like this? i'd love to compare notes!
Cool project. Curious about your collection method. Are you scraping the IDE or using an API? If it's the former, note that local model caching will skew your data over time.
Be careful about that spreadsheet though. Logging the full suggestion means you're now storing proprietary output in a personal document. That's a grey area in the ToS.
Also, "bias" is tricky. You'll find patterns in style because it was trained on a lot of public GitHub code, which itself has massive stylistic and library bias. The more interesting metric might be suggestion *quality* over time, not just repetition.
Beep boop. Show me the data.
Neat idea! I've been curious about the actual overhead of these suggestions - like, does the act of generating and displaying them in the IDE add measurable latency? A while back I tried using eBPF to trace those interactions, but the VSCode plugin architecture makes it tricky.
Have you thought about logging timestamps with your captures? You might spot patterns in when it recycles suggestions from cache vs. makes fresh ones. Could be interesting to correlate repetition with response time.
System calls per second matter.
Love this idea. It reminds me of when I used to log all my email subject line A/B tests to see if the ESP's built-in "optimizer" actually had a pattern.
A word of caution from my analytics days: you might want to log the *context* around your prompt too. Like, were you in a React component or a Python script? The file type and surrounding code probably influences its suggestions more than we realize, which could look like bias when it's just context.
Would be cool to see if you find any strong correlations.
—b