Okay, I know the title sounds a bit harsh, but hear me out. I've been trying to learn about data pipeline tools, and I keep seeing these threads comparing, say, dbt vs. some other SQL transformation tool, or Prefect vs. Airflow vs. Dagster.
Everyone lists features, which is nice, but as a newbie, I'm drowning in abstract points. "Better developer experience," "more extensible," "cleaner UI." What does that *mean*? I need to see it in action.
For example, if we're comparing SQL formatter plugins. Someone will say "Plugin A has more configuration options." Great! But... does it make my 500-line messy BigQuery SQL run faster after formatting? Or does Plugin B actually catch more potential syntax errors specific to BigQuery? That's the kind of comparison I need.
Without concrete benchmarks, these threads just feel like opinion lists. I end up more confused than when I started.
So my question is: why *aren't* benchmarks a standard part of these comparisons? Is it just too much work? Or are the results too dependent on someone's specific setup? I'd love to see something like "for a typical medallion architecture pipeline with 10 models, Tool X took 5 minutes to run locally, Tool Y took 7, and here's the CPU/memory usage."
Maybe I'm asking for too much 😅. But coming from a background where you benchmark database queries, it feels like a missing piece. How do you all actually decide between similar plugins or tools without this data? Do you just spin up a test project for each one? That seems like a huge time investment.
You're totally right, and it hits close to home for me in the CI/CD space. I'll see a thread comparing five different GitHub Actions for setting up a Go toolchain, and it's all "this one's more popular" or "that one has nicer logs." But which one actually caches dependencies effectively and shaves 90 seconds off my pipeline? That's what matters.
It *is* a ton of work, and results are super environment-dependent. My benchmarks on my k8s cluster might not match yours. But you can still share methodology - like, "I ran this Terraform plan 10 times with plugin A and 10 times with plugin B, here's the average time and memory." That's infinitely more useful than "feels more intuitive."
We should demand that. Maybe the first reply in any comparison thread should just be "cool, show us the numbers."
K8s enthusiast