Hey everyone, I just saw the announcement about the new community plugin for Tekton pipelines. As someone still trying to wrap my head around modern CI/CD for data pipelines, this seems really cool, but I'm also a bit lost on the practical side.
I've been using Airflow for orchestration and writing a lot of SQL transformations, mostly in BigQuery. The whole shift to containerized, Kubernetes-native stuff like Tekton feels like a big leap. I understand the concepts—tasks as pods, pipelines defining workflows—but I'm struggling to see how it fits with my daily ETL work.
So, for those of you who have tried it or are more experienced: what's the real-world use case here? Is this plugin something that could help glue together, say, a data validation task, a dbt run, and then a BigQuery load? Or is it more for the application deployment side that I wouldn't touch as much?
Also, how does it compare to just using Cloud Composer (Airflow) for everything? The idea of more fine-grained control is appealing, but I'm worried about it being too complex and me getting overwhelmed setting it all up. Any reviews or simple examples would be super helpful! 😅
Yeah, I'm in the same boat, honestly. I get the containerized tasks idea, but jumping from Airflow to Tekton feels huge.
For your ETL question, I *think* it could glue those steps together. I saw someone mention using it for a dbt test, then a load. But I'm also worried about the setup complexity compared to Cloud Composer. Is it worth that learning curve just for data pipelines?
I'm curious if the new plugin makes it easier to manage, or if it's just another layer to figure out. Have you found any good, simple tutorials for data use cases?
Still learning.