Okay, I'm diving into a subforum that's a bit outside my usual CI/CD lane, but hear me out. This has been bugging me for a while as I watch teams build these elaborate data pipelines only to slap a giant, expensive BI layer on top that just... runs SQL.
Here's my hot take: **If your team writes good, modular, version-controlled SQL and has a decent orchestration tool, you might be overcomplicating things by forcing a full BI platform on everyone.**
Don't get me wrong. I love tools. I'll spend hours tweaking a GitHub Actions matrix strategy. But I see a parallel: sometimes we add tooling for tooling's sake. Many BI tools, at their core, are just SQL query builders with a GUI dashboard renderer. If your data product is consumed by other technical teams or even savvy business users, why not treat your analytics like any other code-driven deliverable?
Think about it:
* Your transformation logic (dbt, plain SQL files) is already in Git.
* Your orchestration (Airflow, Prefect, even scheduled CI jobs) can run these queries on a schedule.
* The output could be:
* Materialized views for direct DB querying.
* Static CSV/Parquet files dumped to cloud storage.
* A simple, auto-generated static site (using something like `dsdocs`) that documents key tables and their refresh schedules.
What you gain:
* **Full traceability:** Every dataset is tied to a commit hash, a pipeline run ID. No more "which dashboard filter produced this number?"
* **Standardized environments:** You promote SQL from dev -> staging -> prod, just like your application code.
* **Cost:** You're trading hefty BI licenses for compute costs, which you can optimize and monitor like any other infrastructure.
Where it falls apart, obviously, is self-serve for *truly* non-technical users. If your business users need drag-and-drop exploration and cannot possibly write a `WHERE` clause, you need a tool. But for embedded analytics, internal reporting for technical stakeholders, or even scheduled report generation? You can build a robust system with what's already in your stack.
```yaml
# Example GitHub Actions workflow for a "BI report" as code
name: Generate Weekly Metrics
on:
schedule:
- cron: '0 9 * * 1' # Monday 9 AM
workflow_dispatch:
jobs:
generate-report:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Run Analysis Query
env:
DATABASE_URL: ${{ secrets.PROD_DB_URL }}
run: |
psql $DATABASE_URL -f ./analytics/weekly_kpi.sql -o ./output/kpi.csv
- name: Publish to Internal Wiki
uses: peaceiris/actions-gh-pages@v3
with:
github_token: ${{ secrets.GITHUB_TOKEN }}
publish_dir: ./output
```
This is essentially a CI pipeline for data. The result is an artifact (CSV, maybe a simple HTML page) available at a predictable URL. It's auditable, reproducible, and cheap.
So my question to the BI experts here: Am I being naive? At what scale or user persona does this "SQL-as-BI" approach completely break down? What specific capabilities of modern BI tools (beyond the SQL generation and viz) are impossible or painfully hard to replicate with a code-first approach? I'm genuinely curious.
pipeline all the things