Spent the last two on-call rotations getting paged for issues that could have been caught by a simple code review. Our team's PR velocity is great, but we're missing subtle bugs in our instrumentation and alerting logic—think mislabeled metrics, incorrect threshold comparisons, or even a typo in a `promql` query that only blows up at 3 AM.
I built a proof-of-concept to integrate Claude Code reviews directly into our GitHub Actions, focusing on our observability stack. The goal: automatically flag potential issues in PRs that touch Grafana dashboards (JSON), Prometheus alerts (YAML), or Terraform for our provisioning. It’s been running for a month and has already saved us from three nasty regressions.
Here's the core of the workflow. It triggers on PRs that modify files in specific paths:
```yaml
name: Claude Code Review
on:
pull_request:
paths:
- 'grafana/dashboards/**'
- 'prometheus/alerts/**'
- 'terraform/monitoring/**'
jobs:
review:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Analyze with Claude Code
env:
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
run: |
# This script gathers the changed files and formats a context-aware prompt
python3 claude_review.py
```
The magic is in the prompt engineering within the `claude_review.py` script. We instruct Claude to act as an SRE reviewer. Example prompt snippet:
```text
You are reviewing changes to observability configuration. Check for:
- Prometheus alert rules: Validate expression syntax, ensure `for:` durations are sensible.
- Grafana dashboard JSON: Check for missing datasource references, broken query templates.
- Terraform: Verify that monitoring module attributes are correctly set.
List any concerns as bullet points. If no issues, state "LGTM for observability".
```
The action then posts the review as a comment on the PR. Key takeaways for a stable setup:
* **Scope it tightly.** Don't review everything—just the code where Claude's expertise adds value (for us, that's config).
* **Set clear, directive prompts.** You get better results by asking for a specific output format.
* **Use it as a first-pass filter.** It doesn't replace human review, but it catches the obvious stuff and lets us focus on higher-level design.
It's like having a junior SRE on every PR, one that never sleeps. Curious if anyone else has tried similar integrations and what pitfalls you've hit—especially around handling large, diff-heavy PRs.
zzz
Sleep is for the weak
This is exactly the kind of late-night pager scenario I'm trying to avoid. The idea of scanning changes to Grafana JSON and Prometheus YAML automatically is a game changer.
I'm curious about the false positive rate. Does Claude ever flag things that are actually correct, like a complex PromQL query it misinterprets? Also, how do you handle the cost of the API calls? Is it per-PR or does it add up quickly across a busy repo?
Interesting approach. I've looked at similar setups but always hit a wall with cost. What's your actual spend for that month of reviews? Are you using a max token limit or specific model to control it?
Also, on the false positive question from user217, I've found the key is in the prompt engineering for domain-specific configs like PromQL. You can't just ask for a general code review. Are you providing Claude with context about your specific metric naming conventions or alerting framework?
Ask me about hidden egress costs.
Great questions. On cost, we're using the `claude-3-haiku` model specifically for its price/performance on this kind of structured review. Our spend last month was under $15 for a repo with about 40 PRs. The trick is a strict `max_tokens` limit and we only feed it the actual diff, not whole files.
You're 100% right about prompt engineering. We gave Claude a "style guide" snippet in the system prompt with our actual metric naming conventions (e.g., `app_http_requests_total`) and a few examples of valid vs. questionable PromQL. It cut false positives way down. Without that context, it would flag every non-standard label as a potential typo.
Your point about feeding only the diff, not whole files, is critical. In my own cost benchmarking, I found that using the full context of a modified Grafana dashboard JSON, even with Haiku, could sometimes triple the token count versus a focused diff. The noise-to-signal ratio in those large JSON objects is just not worth it.
However, I'd add a caveat about the `max_tokens` limit on the *output* side. Setting it too low for the review can cause Claude to truncate its final findings, potentially omitting the most critical issue. It's better to use a more targeted system prompt that instructs it to prioritize and list only the top three high-severity concerns, rather than relying purely on a token cutoff.
The style guide approach is absolutely correct. We've done similar with OpenTelemetry semantic conventions. Without that, you get useless warnings about metric names like `service.cache.hits` when you've deliberately chosen a different naming schema.
You're focusing on the right integration points for observability. The path filter in your workflow is a solid first step, but you should consider adding a severity-based gate.
For example, a change to a critical production alert rule in Prometheus should perhaps block the PR merge until the review is addressed, while a tweak to a developer dashboard might only post a comment. I've seen teams implement this by checking the modified file path against a configurable severity matrix.
Also, echoing the later points on cost, you'll want to immediately add a step to generate a clean, context-aware diff. A raw `git diff` output for a Grafana JSON file includes a lot of noise from internal IDs and positioning fields that Claude doesn't need to see. Pre-process it to strip those out before sending to the API.
show me the SLA