Having observed numerous anecdotal reports of editor slowdowns attributed to AI coding assistant plugins—Copilot, Cursor, Tabnine, and the like—I have found the discourse to be lacking in rigorous, reproducible measurement. Claims of "my editor is sluggish" are insufficient for diagnosis. To address this, I have developed a small, focused benchmark suite designed to quantify the overhead introduced by these plugins during two critical phases: editor startup time and language server request latency.
The suite is implemented in Python and leverages the `psutil` library for resource tracking and subprocess control. It is editor-agnostic in theory but currently targets VS Code due to its extensive plugin ecosystem and CLI controls. The core methodology involves:
* Launching the editor from a clean state (with user data dir and extensions dir controlled) via subprocess.
* Measuring time to main window readiness by monitoring process threads and STDIO output for specific markers.
* Executing a standardized series of language server requests (e.g., completions, hover info, definition) on a known Python test file via the editor's built-in CLI commands where possible, or through automated UI scripting as a fallback.
* Recording peak memory usage (RSS) of the editor process tree throughout the session.
A simplified core of the timing logic is as follows:
```python
import subprocess, time, psutil, json
def benchmark_startup(executable_path, user_data_dir, extensions_dir):
"""Returns startup duration in seconds."""
cmd = [
executable_path,
"--user-data-dir", user_data_dir,
"--extensions-dir", extensions_dir,
"--new-window",
"--disable-workspace-trust",
path_to_test_file
]
start = time.perf_counter()
proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
# Wait for readiness signal (e.g., specific stdout line)
for line in iter(proc.stdout.readline, b''):
if b"Window ready" in line: # Example marker; requires custom extension for precise signal
break
duration = time.perf_counter() - start
proc.terminate()
return duration
```
Preliminary results on my system (macOS ARM, 32GB RAM) using a minimal Python file are revealing. With all extensions disabled, VS Code startup averages 1.2 seconds. Enabling only GitHub Copilot adds approximately 400-600ms to startup and introduces a consistent 80-120ms latency to completion request round-trips. A more complex setup with Copilot, Tabnine, and a full Python language server (Pylance) saw startup times balloon to 3.8 seconds and completion latency variances exceeding 300ms.
I am now seeking to validate and extend this benchmark. I am particularly interested in the following:
* Potential blind spots in the measurement approach, especially for background indexing or network-bound plugin initialization.
* Suggestions for standardizing a "workspace complexity" factor (number of files, node_modules size, etc.).
* Collaborative data gathering across different OSes (Windows, Linux) and editor configurations.
The ultimate goal is to move beyond speculation and provide a framework for users to isolate which specific plugin interaction is causing their performance degradation. The code is available for review and contribution. What are the most critical operational scenarios I should add to the test suite?
Quantifying plugin overhead is a great approach. A clean user data dir and extensions dir for each run is essential, otherwise you're just measuring state from a previous session.
I'd be curious about the audit trail for the benchmark itself. Does your Python script generate a structured log of each run - timestamps, plugin configurations, process IDs, measured intervals? That would let you correlate a specific slowdown with a system event or a background update from the plugin vendor.
When you measure language server request latency, are you isolating the network call from the IDE's own processing? Some of these assistants might be making external API calls. The overhead could be I/O wait, not CPU, which would show up differently in your psutil metrics.
Logs don't lie.
Great points, especially about clean directories - that's a must. I didn't see any audit log in the OP's description, which is a huge gap. Without timestamps and config hashes, you can't tell if a spike was a plugin update or just your antivirus scanning.
On your network point, you're spot on. My own tests show the latency often *is* the API call. You could run a local proxy to split the timings - measure the round-trip from editor to proxy (plugin overhead) and proxy to external service (network). `psutil` might miss that I/O wait unless you're tracking specific process states.
Prompt engineering is the new debugging
Your methodological focus on quantifying startup and request latency is precisely what's needed to move beyond anecdotes. However, I would propose expanding the latency measurement to differentiate between local plugin processing and remote API dependency. For instance, when measuring language server request latency, you could instrument a local loopback proxy to capture the timing of the outbound HTTP call separately from the editor's own processing thread. This would isolate network I/O wait, which psutil's CPU-centric metrics might obscure. A spike in the proxy timing would indicate external service slowness, not inherent plugin overhead, fundamentally changing the interpretation of the results.
Measure twice, cut once.
This is exactly the kind of tool I've been looking for! I'm trying to decide between a couple of these AI plugins, but the performance stories are all over the place. Measuring startup time and request latency is a great start.
A question, though: how are you handling the actual language server requests in a standardized way? Is there a specific set of operations you run on the test file? I'm just wondering if the benchmark might miss overhead from certain features, like in-line chat or code actions, that could also cause slowdowns.
Will you be publishing the results anywhere? I'd love to see how the plugins stack up.