Hey everyone, I've been lurking for a bit but this is my first real post. I'm working on my first major pipeline project at my company (moving from batch scripts to Airflow), and I'm hitting a wall I didn't see coming.
We're evaluating a fancy new cloud ETL tool, and my boss is totally focused on the per-hour runtime cost. But from what I'm building, I'm spending maybe 80% of my time just getting the data *ready* to be processed. I'm talking about:
- Writing custom Python functions to clean and standardize wildly inconsistent source files.
- Figuring out how to handle late-arriving data or schema drift in our raw zone.
- Building all the error handling and retry logic for API calls that fail randomly.
The actual transformation logic, which the tool license is for, feels like the smallest piece. I'm worried we'll buy this shiny tool and then my team will still be drowning in prep work, making the ROI look terrible.
Is this normal? I keep hearing "garbage in, garbage out," but nobody talks about the "garbage preparation" tax. How do you all account for these hidden data prep labor hours in your TCO models? Do you just bake in an estimated percentage for "data janitor" work, or is there a better framework?
I'd love to hear how more experienced teams track or even reduce this. Are there specific practices with dbt or data lake architectures that help contain this cost?
Feeling a bit overwhelmed 😅.
-- rookie
rookie
You've hit on a classic, unaccounted-for cost center. The tool's runtime fee is visible, but the engineering hours to create a clean, reliable ingestion layer are almost always an order of magnitude higher. I've seen this exact scenario play out with teams adopting cloud data warehouses.
In my experience, the TCO model fails because it assumes the data arriving at the tool's input port is already "tool-ready." Your "garbage preparation tax" is real, and it's largely fixed labor cost, not variable cloud cost. You can't just add a percentage. You need to itemize the ongoing maintenance for each source system: schema change monitoring, API failure handling, and data quality checks. This becomes a permanent headcount requirement.
Pushing back requires quantifying your current "data janitor" hours and projecting them forward. Show that the expensive runtime will be idle if the prep work isn't funded. The tool may still be worth it, but only if the business understands it's buying an engine, not the entire fuel refinery.
SQL is not dead.
Spot on about the headcount requirement. That's the part finance often misses. The runtime cost scales with usage, but the data prep cost scales with the number of source systems. Adding a new SaaS vendor to your pipeline means another API integration, another set of schema quirks, another set of alerts to monitor. That's a permanent engineering burden, not a one-time project cost.
The "buying an engine, not the fuel refinery" analogy is perfect. I've found success framing it as a support ratio: one platform engineer can support X clean, stable sources, but only Y messy, volatile ones. If the business wants to ingest from ten unstable APIs, they're implicitly budgeting for two engineers, not just the tool license.
Keep it civil, keep it real