Okay, I’ll admit I’m still pretty new to this whole data engineering world, but I’ve been hearing this phrase "single source of truth" everywhere – in our Consensus reviews, in tool marketing, in my team meetings. We’re implementing a new data platform and leadership keeps saying that’s the goal.
But after my first few months trying to build pipelines (mostly with Airflow and some basic dbt), I’m starting to think it’s kind of a myth? Or at least, the way it’s promised feels like one.
For example, we have a core "products" table in our data warehouse that’s supposed to be the SSOT. But then I found out the marketing team maintains their own product taxonomy in a separate spreadsheet for campaign tracking, because our official table doesn’t have all the attributes they need. And finance has another version with slightly different cost allocations. So we have three "truths" 😅.
My question for you all is: is the "single source of truth" more of an aspirational guideline than an actual achievable state? Should we be aiming for a "single source of *data*" but accepting that transformed versions for different departments will always exist? I’m trying to understand best practices here before I build more pipelines that might be based on a flawed assumption.
I’d love to hear about your experiences, especially if you’ve worked in environments that claimed to have achieved this. What does it *really* look like on the ground?
-- rookie
rookie
Yeah, it's definitely aspirational. In my experience, the real goal is to make your core data reliable *enough* that people *want* to use it.
Those spreadsheets pop up because the official source isn't serving a need. Can you extend that core products table with those extra attributes? Sometimes a "single source" needs to be more of a central hub that can flex for different teams.
The spreadsheet sprawl hits cloud costs too. You see it when teams start tracking their own spend in Google Sheets because the central dashboard doesn't break it down the way they need.
I completely get that. "Reliable enough that people want to use it" is such a good way to put it.
From a project management angle, I've seen the same thing happen with team scheduling. The official resource plan becomes the "source of truth," but if it's too rigid, leads will just keep their own shadow spreadsheets for daily assignments. The trick is making the central plan flexible enough to absorb those real-time changes.
Maybe that's the real single source, a system that's actually useful for day-to-day work, not just for reporting.