Hey everyone, I’m still pretty new to setting up data pipelines and their observability, so please bear with me. I came across this idea and it kind of blew my mind because it seems so obvious now.
I was reading this thread title and it got me thinking about my own small project. I set up Airflow with the default logging config and connected it to BigQuery and a logging service. My bill for the logging part was way higher than I expected after just a week! 😅 When I dug in, I realized that the DEBUG level logs from every single task run were making up like 70% of the volume. Most of it was just internal scheduler chatter and super granular step-by-step output I never look at unless something is *really* broken.
So my question is: Is this really a universal hot take? It seems like a "well, duh" thing, but I'm guessing in big production environments it's trickier. Are people actually doing this systematically? Like, do you have a rule to never run DEBUG in prod, or do you use sampling? I'm worried I might turn them off and then miss something crucial during an incident.
Also, for tools like dbt or Airflow, where do you even configure the default log level? Is it in the environment variables, the config files, or inside the pipeline code itself? I'd love to hear how more experienced folks handle this practically.
Yeah, been there. The default logging for a lot of these tools is way too chatty for production.
Your worry about missing something during an incident is valid. A common compromise is to keep DEBUG on a small percentage of your runs via sampling, or only enable it for a specific task or module when you're actively troubleshooting. That way you keep the volume down but can turn on the firehose when you need it.
For Airflow specifically, you'd set the log level in the config file, usually `airflow.cfg`. Look for the `logging_level` setting. But I'd also check the config for the actual task runners, like the Celery workers, as they can have their own defaults.