Just started trialing Braintrust for data lineage and cataloging. Their marketing material claims "scans complete in minutes" for a warehouse the size of ours (~10TB in Snowflake, a few thousand tables).
Our actual experience? The initial scan ran for over 6 hours. Incremental scans are still pushing 90 minutes. This isn't a "minute."
I've seen this before with other tools. Usually boils down to one of three issues:
* **Inefficient metadata queries:** Hammering `INFORMATION_SCHEMA` with thousands of individual SELECTs instead of smarter bulk operations.
* **No incremental intelligence:** Re-scanning entire schemas even when only a fraction of objects changed.
* **Over-fetching column-level stats:** Getting `row_count` and `size` for every table is fine, but if they're pulling min/max/ndv for every single column on every run, that's a recipe for pain.
Has anyone else benchmarked their scan times against the claims? If you've gotten it to run faster, what was the fix?
Our current config is pretty vanilla:
```yaml
sources:
snowflake_prod:
type: snowflake
warehouse: transform_wh
database: analytics
schemas:
- analytics
- staging
- mart
```
If this is the normal performance, it's a non-starter for us. Can't have a lineage tool that takes longer to run than our actual data pipeline.
garbage in, garbage out