You're so right about logging both the immutable hash and the pipeline reference. That's saved my skin more than once. We treat the hash as the source of truth for the exact experiment, but the pipeline reference is what we actually use to debug when something upstream breaks.
> A small shared library or logging wrapper saves you there.
This is the key. We made a tiny Python package that enforces a `data_snapshot` dictionary in Langfuse's metadata, with required keys for `hash` and `feature_store_ref`. It's five functions total. Without that, we'd have `data_hash`, `snapshot_id`, and `commit_sha` all for the same thing by now.
The wrapper also auto-generates a link to our data catalog if you give it a pipeline version, which is a nice little UX win for the team.
Keep deploying!
Welcome to the mess of moving beyond spreadsheets, it's a great problem to have!
On your business parameters point, I'm with user938. That tiny shared wrapper library is the secret sauce. It gives you the freedom of Langfuse's metadata for quick "added weather data" tests, but bakes in the required fields like `lookback_window` so it doesn't become a junk drawer. You get the best of both.
For comparing holiday seasons, the real question is where your team lives. If they love notebooks and pulling data into Grafana, the query APIs are comparable. If they want to quickly click around in a built-in UI without writing a line of code, MLflow's charts have a clearer advantage.
Does your team have the appetite to maintain a small, shared logging package? That honestly decides which tool's philosophy works better for you.
measure twice, ship once