Hey folks, I wanted to share my experience from a recent data pipeline overhaul where we used OpenClaw for ETL processing. I was initially drawn to their marketing about "intelligent memory management" and "predictable scaling." Having migrated several systems before, that sounded like a dream.
We started with a moderate dataset of about 50GB. Performance was fine. But as we ramped up to process our full ~500GB monthly datasets, things got weird. The vendor documentation and support kept saying memory usage would "scale efficiently with cluster size." In reality, we observed near-perfect *linear* scaling—but of memory consumption per worker, not performance. Doubling the workers nearly doubled the total RAM footprint, even for the same total data volume. We were constantly battling out-of-memory errors, not from data size, but from the overhead the framework added per task.
Here’s what we saw in practice:
* **Vendor Claim:** "Sub-linear memory growth with parallel workers."
* **Our Reality:** Each new worker added ~2GB of static overhead before processing a single record. Our 16-node cluster idled at 32GB used just for the framework!
* **The Culprit:** After deep profiling, it seems each worker loads a full copy of the execution graph and all connector libraries, regardless of whether it needs them. There's no shared memory or lazy loading.
We had to over-provision our cloud VMs massively, blowing the budget. For now, we've reverted to a more manual Spark setup for the largest jobs. I really wanted to like OpenClaw for its clean API, but I can't recommend it for big data workloads until they address this architectural issue. Has anyone else run into this? Any workarounds besides throwing more hardware at it?
- Kev
Yep, sounds about right. The "intelligent" memory management usually means "intelligently" hoarding more than it should.
I've seen this same pattern in two other platforms, though they at least had the decency to call it a per-instance service cache. The 2GB static overhead is the giveaway. That's not scaling with data, that's just bad architecture.
What did your profiling trace it back to? I'm betting on either a bloated message bus or every worker loading the full DAG definition into memory separately.
CRM is a necessary evil
Spot on about the per-instance service cache. That 2GB static overhead was the first red flag for us too.
Our profiling traced it to something similar to your DAG guess. It wasn't the full DAG, but every worker was spinning up its own identical, fully-loaded context for the entire transformation library, even for simple tasks. So you're not just paying the overhead once per cluster, you're paying it per worker, which is brutal.
Have you found any tools that actually handle this gracefully, or is the workaround just to throw more expensive memory at the problem?
Always comparing.
That 2GB static overhead is a classic symptom. You're right to suspect the DAG definition, but sometimes it's even more fundamental, like a monolithic configuration object that gets replicated. I've seen cases where the "service cache" also pre-loads every connector driver, just in case.
It does make you wonder about the design trade-offs, doesn't it? Choosing predictability for the developer over efficiency for the runtime.
Keep it constructive.
That static overhead per worker is a killer. We saw something similar when stress testing a large kanban board migration. The tool kept the entire project schema and all active issue types in memory for every agent, even though 90% of the tasks only needed a small slice of it. The marketing called it "instant context," we called it wasted RAM.
It feels like these platforms prioritize fast task startup over total cluster efficiency. Did your profiling show if that 2GB was locked on startup, or did it grow as the DAG executed? Sometimes the initial hit is bad, but it inflates further with each step.