Has anyone tried using ChatGPT as a second-tier reviewer in their code CI pipeline?

We experimented with this. The false positive rate was too high for our team.

The main issues:
* It's decent at spotting SQL antipatterns (like missing indexes hints in Snowflake) if you prompt it with your schema. But it hallucinates constraints.
* For Airflow best practices, you're better off writing a custom Pylint plugin. It's more reliable.
* Token limits force you to chunk files. Context gets lost, especially in DAGs.

You'll spend more time vetting its comments than you save.

Totally feel you on the hallucinated constraints. I ran a test where it flagged a "missing foreign key" in a table that was meant to be a staging layer. The model was confidently wrong.

You're right about the custom linter for Airflow - we did that for `execution_date` vs `logical_date` and it works perfectly. For token limits, I found feeding it just the task definitions and skipping the boilerplate helps a bit, but it's still messy.

It's become a tool for junior devs to get *started* thinking about review comments, but we'd never gate a pipeline on it.

You're asking the right questions. The problem isn't the AI, it's the expectation.

> catch subtle logic errors in SQL
It can't. It doesn't understand your business logic or data lineage. It'll guess based on public repos, which often leads to spectacularly wrong "best practice" suggestions for your specific warehouse.

You identified the real cost: another complex pipeline to maintain. The setup and prompt engineering to make it even marginally useful will eat the time of your most senior person. That's the opposite of helping an overwhelmed team.

Your existing linters are deterministic. Focus on extending those rules for your Airflow and SQL patterns. That's a boring, permanent fix.

I ran a whole experiment with this exact setup last quarter, feeding our Airflow DAG diffs to GPT-4. Your hunch is right - the token limit is a killer. You end up playing this weird game of "what's the most crucial snippet?" and lose all the imports and context.

For your Airflow example, it did sometimes flag `datetime.now()`, but then it would also weirdly criticize perfectly valid uses of `Variable.get`. It's like a junior dev who's read a blog post and applies it too broadly.

My takeaway was similar to others - it's okay as a pre-review checklist generator for the author, but as a gatekeeper? It adds friction. The prompt engineering to make it useful for your specific data patterns (like our custom BigQuery macros) became a part-time job. Honestly, extending our SQLFluff config with a couple custom rules was less exciting but solved the problem for good.

I actually ran this for three months on our data team's PRs using the OpenAI API in a GitHub Actions step. The prompt engineering alone was a massive time sink, and that's the hidden cost nobody talks about.

You asked about structuring prompts. You end up writing a mini-rulebook for the LLM every time, like:

```
You are reviewing a changed Airflow DAG. Ignore imports and boilerplate.
Focus only on added or modified tasks.
Check for: use of datetime.now(), missing catchup=False, direct SQL string formatting.
Do NOT comment on variable naming.
```

But then it completely misses a chain of tasks where the order creates a deadlock, because you had to leave out the surrounding context to fit tokens. It'll catch the low-hanging fruit you could have written a simple regex for, and miss the complex stuff you actually need help with.

For SQL, it constantly suggested using `QUALIFY` in Snowflake when our version didn't support it yet. It's confidently wrong on syntax and features specific to your environment.

It added about 45 seconds to every pipeline run and generated so much noise that the team started ignoring the comments entirely. We turned it off and invested that time into writing a few targeted SQLFluff custom rules instead. Wasn't worth it.

You're right to focus on that token limit. I ran into the same issue trying to feed entire DAGs - you have to strip out imports and boilerplate to get under the limit, but that strips out the dependencies between tasks, which is where the real review needs to happen.

> flag potential performance issues in BigQuery/Snowflake SQL
It's surprisingly bad at this for complex queries. It'll suggest a trivial `SELECT *` fix but miss a missing partition filter on a 10TB table because it lacks your actual table metadata and cardinality. The cost isn't just the API calls - it's the hours you'll spend tuning prompts to reduce noise, only to find the signal is weak anyway.

Your instinct about it becoming another complex pipeline is spot-on. We ended up replacing it with a few targeted SQL and Airflow rules in our existing linter config - boring, but it runs in seconds and never hallucinates a constraint.

Exactly. The hidden cost is you're now maintaining a linter written in English prose, with non-deterministic output. You traded a static config file for a prompt that needs versioning, testing, and a human to interpret its hallucinations.

And that "confidently wrong on syntax" is the real killer. It reviews based on its training data, not your actual environment. You spend time correcting its misconceptions instead of reviewing the actual code.

So you're paying API costs to generate the problems a custom linter solves for free, with less reliability. The irony is thick.

Oh, you're right on the edge of a time sink. Everyone here has already nailed the big issues: tokens, hallucinations, prompt engineering becoming a job itself. Let me add a specific, practical example from our failed attempt that relates directly to your points.

You asked about structuring prompts for consistency. We tried exactly that, feeding diffs of Airflow DAGs. We wrote a novel of a prompt specifying the exact patterns to look for, like `datetime.now()` and `catchup=False`. It would indeed flag those, sometimes. But then it would also, completely unprompted, suggest replacing a `PythonOperator` with a `DockerOperator` for "better isolation," adding a massive architectural change we never asked for. You're not just fighting for correct feedback, you're fighting to stop it from inventing new problems. The maintenance isn't just tuning the prompt for what you want, it's constantly updating it to *stop* it from doing things you don't.

And on SQL performance? Forget it. It once flagged a `SELECT column_a, column_b` as "inefficient" and suggested using `SELECT *` for "better readability," completely ignoring the 30-column wide table and the actual performance hit of pulling unnecessary data. The signal-to-noise ratio is brutal, and the noise is often dangerously wrong. You'll spend your review time debunking the AI's suggestions instead of thinking about the actual logic.

Yep, the unsolicited architectural advice is the worst. It's like a reviewer who can't stay in their lane.

We saw it suggest moving a simple, stable `PythonOperator` to a full `KubernetesPodOperator` "for scalability," adding a ton of YAML boilerplate and cluster permissions for zero benefit. It doesn't understand the operational cost.

Your SQL example is perfect. It optimizes for textbook readability, not the actual data volume or cost. You end up babysitting it more than it helps.