Hooking the API directly to Metabase is a clever workaround for the over-engineering problem, and I've done it for smaller projects. The historical data question is the killer, though. Metabase will only pull what's currently in the API, which is usually just the current state of items.
You end up having to implement some form of persistence anyway, even if it's just a daily cron job dumping the API JSON into a PostgreSQL table that Metabase reads from. At that point, you're just building a narrow, purpose-built warehouse, and you've traded maintaining Airbyte for maintaining your own sync script and schema migrations.
Automate everything. Twice.
Exactly. That daily cron job sync is what we run. It's a simple python script that dumps Runway's board JSON into a timestamped S3 folder.
Metabase points at a view that flattens the latest JSON dump. It's maybe 50 lines of code total. The schema drift risk is there, but it's less work than managing a full ETL pipeline.
—cp
Your approach is sensible for immediate reporting needs, but it introduces a silent risk with that "latest JSON dump" view. You're effectively building a slowly changing dimension table but discarding the history.
If a field is renamed or a card type is retired in Runway, your flattening view will break or, worse, present incorrect data as if it always looked that way. A more durable pattern is to write each dump to a separate table with an extraction timestamp and have Metabase query a version-aware union. That 50-line script becomes 150 lines, but it preserves your ability to track state changes over time, which is often the entire point of the historical data exercise.
—BJ
You've identified the core schema evolution problem with Type 2 dimensions in ad hoc syncs. We hit this exact issue when a "client_status" field was split into "contract_status" and "payment_status." Our latest JSON view silently merged them, making historical trend analysis on the original field impossible.
The version-aware union pattern is correct, but the 150-line script estimate is optimistic if you want to handle soft deletions or capture relationship changes between cards. We ended up storing both the raw JSON blob and a flattened, versioned table, which allowed us to rebuild historical snapshots after the fact using dbt.
Without that, you're just building a log of point-in-time inaccuracies.
Your data is only as good as your pipeline.