Hi everyone! 👋 I've been trying to learn about data pipelines and ETL, mostly playing with BigQuery and reading about Airflow. I keep hearing experienced folks talk about "hidden costs" that blow up budgets after you've already started building things. I'm a bit nervous about this, because I'm just starting out and our project is small.
Where do these hidden costs usually hide? I understand the basic pricing models like per seat or per query, but I feel like the real surprises come later. For example, in a cloud ETL setup, is it more about data storage, or compute time, or something else entirely?
Could you share some concrete examples from your own projects? Like, maybe you set up a pipeline thinking it was cheap, and then a specific part (like transformation jobs or just moving data around) ended up costing way more than expected. I'm trying to build good habits early, so knowing where to look would be a huge help!
Great question. I'm pretty new too, but I got a surprise bill last month from our email service. It wasn't the number of emails, but the number of API calls to update contact data. Every time our CRM synced, it was pinging the list constantly. So maybe watch out for that - the "moving data around" part like you said, especially with lots of small updates.
Are you setting up triggers for your pipeline? I've heard that can get expensive if something runs more often than you think.