Just built a CLI wrapper to log all Aider suggestions for audit.

I've been using Aider consistently for several months now, primarily for refactoring tasks and generating boilerplate code. While I appreciate its utility, my background in experimentation and analytics has made me increasingly uneasy about the opacity of its decision-making process. Specifically, I found myself wondering: on which suggestions does it succeed, and on which does it fail? What patterns exist in the rejected edits? Without systematic logging, any review of Aider's performance is anecdotal at best.

To address this, I've built a lightweight CLI wrapper that intercepts, timestamps, and logs every suggestion Aider makes, along with the final user action (accept, reject, edit). The goal is to create an audit trail for later analysis, enabling a data-driven assessment of its strengths and weaknesses across different types of coding tasks. The core functionality involves wrapping the `aider` command and parsing its streaming output.

Here is the core of the wrapper script (a Python script that uses `subprocess` and `sys`):

```python
#!/usr/bin/env python3
import subprocess
import sys
import json
import time
from datetime import datetime

LOG_FILE = "./aider_audit.log"

def parse_suggestion_block(block):
"""Heuristic to extract the suggested code change from Aider's output."""
# This is a simplified parser; may need refinement for edge cases.
lines = block.split('n')
suggestion = {"original": [], "updated": []}
in_suggestion = False
for line in lines:
if line.startswith('@@'):
in_suggestion = True
continue
if in_suggestion:
if line.startswith('-'):
suggestion["original"].append(line[1:])
elif line.startswith('+'):
suggestion["updated"].append(line[1:])
return suggestion

def main():
cmd = ["aider"] + sys.argv[1:]
process = subprocess.Popen(
cmd,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
text=True,
bufsize=1
)

current_block = []
logging_entry = {}

for line in process.stdout:
sys.stdout.write(line)
sys.stdout.flush()

# Detect the start of a suggestion (simplified trigger)
if "Do you want to apply this edit?" in line or "Show diff" in line:
if current_block:
block_text = ''.join(current_block)
suggestion = parse_suggestion_block(block_text)
logging_entry = {
"timestamp": datetime.utcnow().isoformat(),
"suggestion": suggestion,
"files_involved": [], # Would need to extract from context
"user_action": None # To be filled post-prompt
}
current_block = []
current_block.append(line)

# Detect user response (Y/N/A)
if line.strip() in ['Y', 'y', 'N', 'n', 'A', 'a']:
logging_entry["user_action"] = line.strip().upper()
with open(LOG_FILE, 'a') as f:
f.write(json.dumps(logging_entry) + 'n')

process.wait()

if __name__ == "__main__":
main()
```

The log entries are written as JSONL (JSON Lines), which is ideal for later analysis. A sample entry looks like this:

```json
{"timestamp": "2024-07-15T14:22:05.123456", "suggestion": {"original": [" print('Hello, World!')"], "updated": [" print('Hello, Aider!')"]}, "files_involved": ["hello.py"], "user_action": "Y"}
```

My immediate next steps are to enrich the metadata captured and build an analysis pipeline. Planned enhancements include:
* Extracting the explicit file names being modified from Aider's context.
* Logging the initial user prompt/request that triggered the suggestion.
* Capturing the state of the git diff before and after the session.
* Adding a simple classifier for the type of change (e.g., "refactor", "add feature", "fix bug", "write test").

Once I have a sufficient dataset (I'm aiming for n>500 suggestions), I intend to run a formal analysis focusing on:
* **Acceptance Rate:** Overall and segmented by change type.
* **Error Rate:** Instances where an accepted suggestion led to broken code or required immediate manual correction.
* **Session Progression:** Whether acceptance/rejection patterns change within a single coding session.

I'm sharing this here to solicit feedback on the approach and to see if others in the community have attempted similar measurement initiatives. What other metrics would be valuable? Are there pitfalls in my logging methodology I may have overlooked? The broader objective is to move beyond subjective impressions of AI coding assistants and towards a more rigorous, cohort-based evaluation framework.