Hey everyone. I've been lurking here for a while, but this felt like a big enough shift to finally post about. My team just made a major pivot in our automation stack, and the emotional payoff was immediate.
For the last year, we were using a popular, powerful workflow orchestration tool (let's call it "Claw" for the sake of the story). It was great on paper—feature-rich, scalable, the whole deal. But for our mid-sized team and relatively straightforward daily batch pipelines (mostly BigQuery transforms, some API pulls), it felt like overkill. We spent more time wrestling with Claw's abstractions, debugging its scheduler, and worrying about YAML configs than on our actual data logic. Every deployment felt like we were one misconfigured sensor away from breaking a production DAG.
So last month, we proposed an experiment: replace one complex Claw DAG with a simple, scheduled Python script in a Docker container. The goal was just to see if it was viable. We used Prefect Core for very lightweight task retry and logging patterns, but the core logic is just plain Python.
Here's a simplified version of the skeleton we followed:
```python
import logging
from prefect import task, Flow
from datetime import datetime
# ... our actual BigQuery client calls, pandas ops, etc.
@task(max_retries=3, retry_delay=300)
def extract():
# Simple API call or SQL query
return data
@task
def transform(data):
# Pandas or just SQL string building
return transformed_data
@task
def load(transformed_data):
# Load to BigQuery
job = client.load_table_from_dataframe(...)
return job.result()
with Flow("daily_etl") as flow:
e = extract()
t = transform(e)
l = load(t)
# Then we run it on a schedule via a simple K8s CronJob or Airflow (just to run this script).
```
The result? That pipeline is now the most reliable one we have. The code is in a file we can read and debug directly. The team's stress around "orchestration magic" has plummeted. We're now migrating other pipelines to this pattern.
I'm left wondering: for those of you with similar stacks (Airflow or similar for overall coordination, BigQuery, dbt for some transforms), where do you draw the line? When does a heavyweight orchestrator become more burden than benefit, and what's the sweet spot for scripted automation? Are we just kicking the can down the road and will regret this when we need complex fan-out dependencies?