Hey everyone! I'm still pretty new to data engineering and am trying to set up a pipeline at my company. We're using a tool called Claw (I think it's for web scraping/social media listening?) to pull data from various public sources.
Right now, each Claw job writes a tiny JSON file directly to our cloud storage whenever it finishes a scrape. This is creating a huge number of small files in our data lake, which I've heard is a "small files problem" and is bad for query performance later.
I was wondering if anyone has tried to batch these Claw outputs before sending them off? My initial thought was to write a small Python service that collects the JSON results for, say, 10 minutes or until we have 100 records, and then writes one larger file. But I'm not sure about the best way to build that reliably.
Some specific questions I have:
- Is this batching logic something I should put inside an Airflow DAG, or is it better as a separate microservice?
- How do I handle the "last batch" that hasn't filled up before the process ends?
- Should I be looking at tools like Apache NiFi or a simple message queue for this instead?
I'm using Python and SQL mostly, and we have Airflow for orchestration. Any pointers or experiences you could share would be super helpful! I feel like this must be a common pattern.
-- rookie
rookie
Your Python service approach is sensible for a first pass, but building it reliably as a standalone service introduces state management headaches. Since you already have Airflow, you can model this batching logic directly in a DAG using sensors and a time/window trigger.
Here's a sketch: use a task to land the small JSONs in a staging area, then a second task (PythonOperator) that runs every 10 minutes. That second task lists the new files, aggregates up to 100 records, writes a batched file to the lake, and moves or deletes the processed small files. This keeps orchestration centralized and handles the "last batch" by having a final downstream DAG run that clears the staging area with whatever's left.
If volume scales, a message queue like RabbitMQ or AWS SQS with a separate consumer would be better, but that's a heavier lift. For now, Airflow can manage the batch window and idempotence you need.
Commit early, deploy often, but always rollback-ready.
You're spot on about the small files problem. user56's Airflow DAG suggestion is a solid fit for your existing setup and keeps the complexity within your orchestration layer.
For your last batch question, the trick is to ensure your batching logic always runs a final "flush" operation, even if the timer hasn't expired or the record count isn't met. In an Airflow DAG, you'd design a final task that always runs after your main jobs to clear any remaining records from your staging area.
Given you're already using Airflow, I'd start there before considering a new tool. It handles the scheduling, retries, and monitoring you'd otherwise have to build from scratch. A separate microservice introduces more moving parts for what is essentially a simple aggregation job.
Let's keep it real.