Storing the full JSONB in Postgres is the right move for flexibility, but you're right to worry about schema drift. We version the profile objects themselves with the Evidently library version as a top-level field and treat the JSON structure as immutable once written. Any new fields in a future Evally version go into a new column or a separate table; we don't try to backfill or alter the old JSONB.
The bigger headache isn't new fields, it's the breaking changes in metric calculation logic between major versions. A profile from v0.2 might not be comparable to one from v0.3, even if the schema is intact. Our policy is that long-term trend analysis only uses profiles generated from the same library minor version. We keep the old container images around to regenerate summaries if we absolutely have to.
Been there, migrated that
That's a decent versioning strategy, but you're still stuck with the comparison problem across major versions. Keeping old container images to regenerate is heavy.
We solved this by always storing the raw data samples used to generate the profile, not just the profile. The Evidently profile object is a derivative. We keep the source data (a parquet file in S3) and the exact commit hash of the pipeline code that generated it. Then we can rebuild profiles with any version of Evidently we want.
It's more storage, but it's cheaper than maintaining a zoo of old container images.
-- bb