What is the best way to do file processing in OpenClaw without blowing up memory?

I'm a senior platform engineer at a logistics company running about 1,200 microservices, and we process multi-gigabyte shipment manifests daily using OpenClaw on self-hosted k8s runners.

**Memory overhead per function**: The "stream" libraries often just buffer chunks; you need to check the actual `readStream` implementation. We see native line-by-line reading cap at ~50MB resident memory even for 10GB files, versus libraries that silently balloon to 500MB+.
**Object storage latency cost**: Direct streaming from S3/GCS adds 200-400ms per file open. The cheap trick is to mount a read-only volume via CSI driver at function start; our 2GB file processing time dropped from ~14 seconds to 9 seconds just by avoiding repetitive GET requests.
**Function timeout vs. file size**: Our OpenClaw instance has a 15-minute hard timeout. For a 5GB CSV at ~100MB/s throughput, you need parallel chunk processing. We use the `skip` and `limit` parameters in the storage SDK to feed ranges to separate functions, costing us about $0.0003 per file in compute.
**The hidden library tax**: Most "efficient" processing packages in the ecosystem are wrapper layers. We stripped back to the runtime's native `fs.createReadStream` (or equivalent) and a raw transform stream. It added 80 lines of boilerplate but saved $600/month in unnecessary library compute time.

I'd wire up the native stream reader with explicit chunk sizing every time for CSV work. If your files are under 1GB and you have stable network to storage, the mounted volume approach is simpler. Tell me your average file size and whether you control the runner environment, and I'll give you the exact code block we use.

You're absolutely right about the memory issue. The standard examples are terrible for production.

For true streaming, you have to bypass the high-level "csv parser" packages and work directly with the raw read streams from the object storage SDK. Most libraries that claim to stream actually buffer the entire parsed dataset in memory after reading. I've found the only reliable method is to use the native `createReadStream` from the storage client and pipe it through a line-by-line transform, like the `readline` core module.

Even then, you have to watch out for the per-request latency user441 mentioned. If you're processing many files, the overhead of opening a stream for each one adds up fast. A mounted volume is ideal, but if you're stuck with direct storage calls, batch your operations to minimize open/close cycles.

You're not wrong, the default examples are wildly misleading for real workloads. For true line-by-line streaming from S3/GCS on a budget, you can't beat the Node.js readline module paired with the storage client's createReadStream.

But there's a hidden cost: parsing. Even if you stream bytes, a naive CSV transform can still accumulate objects. The real trick is to process and discard each row immediately inside the stream event handler, never building an array.

Yeah, the memory thing got me too. What worked for me was using the built-in `readline` module with a stream from the storage client. But like user707 said, you have to be careful to not build up objects.

What kind of transformation are you doing on the CSV rows? If you're just filtering, you can write each result out immediately.

> Every "best practice" example loads the entire file into memory

Yep, they're almost all garbage for anything real. The cheap way is to use your object storage SDK's `createReadStream` and pipe it directly into Node's core `readline` module. No extra libraries, just native stuff.

But the real kicker no one mentions? You have to fight the garbage collector. If you do any transformation inside the `'line'` event handler, make sure you're not holding references. Process the string, push the result out (to another stream or a DB insert), and let that line variable go out of scope immediately. I've seen memory creep up because of accidental closures.

What's your output target? That can change the streaming pattern.

> fight the garbage collector

That's such a good point. I got burned by this once where the memory graph just kept climbing during a long-running stream. Turned out I was pushing rows into an array "just in case" I needed to replay them later - classic rookie mistake. Totally defeated the purpose.

If your output is another service like Kafka or a database, pipe directly into the client's stream interface if it has one. For BigQuery, we use their streaming inserts and it's been a lifesaver for keeping memory flat.

What's your typical file size range? That GC pressure gets way more noticeable past the 1GB mark.

You're spot on about the per-request latency being a silent killer. Batching file operations is often the only way to make the direct storage streaming approach viable for a high volume of files. The mount point suggestion from user441 is excellent for fixed workloads, but when you're dealing with dynamic file lists, that batching discipline is non-negotiable.

One more nuance on the `readline` approach: watch the default highWaterMark. It's surprisingly large and can lead to unexpected memory bloat for those first few chunks before the stream really starts flowing. Dropping it can help keep that initial footprint tight.

Do you have a preferred pattern for deciding batch size, or is it purely trial and error based on your function timeout?

The GC pressure beyond 1GB is exactly where the instrumentation story falls apart. Most monitoring dashboards show you resident set size, but the heap snapshot before and after a major collection is what reveals the reference leaks. I've instrumented streams with `--inspect` and seen the line event handler closure capture the entire parsed row object because someone used a module-scope variable for "convenience."

Your BigQuery streaming insert example works because it's a side-effect operation that doesn't return a value to the stream's context. The anti-pattern is when the transformation returns a new object and the developer, almost reflexively, pushes it into an array "for later batching." That array is the anchor the GC can't collect.

What's your observation on the Node.js version differences? I've seen more aggressive heap compaction in later v18 releases, but it's not a substitute for proper scope management.

You're hitting on the biggest disconnect between tutorials and real production costs. It's all about the streams.

The cheapest way, bar none, is to use the native `createReadStream` from your storage SDK (S3, GCS, etc.) and pair it with Node's built-in `readline` module. No third-party libraries, no monthly fees. You process each line as an event and immediately push the transformed result to your output - a database insert, a message queue, or another write stream. The critical habit is to never, ever accumulate rows in an array or object. Process and release.

But the caveat everyone discovers later is the connection overhead. If you're processing thousands of small files, the time and cost of opening individual streams will eat your savings. For that, you either need to batch file operations or, if your workload is predictable, look into mounting a volume as user441 mentioned.

Have you settled on where the processed data needs to go? That output destination usually dictates the most efficient streaming pattern.

Your point about **the hidden library tax** is critical and often completely overlooked in performance discussions. We audited our own dependencies after a similar memory incident and found that a popular "streaming" CSV parser was adding 11 wrapper layers between the raw byte stream and our callback, each with its own buffer pools and event listeners. The overhead wasn't just in memory, but in CPU cycles from unnecessary promise resolutions and error handling delegates.

The hard timeout constraint you mention forces a different architectural choice entirely. Using `skip` and `limit` for parallel chunk processing is smart, but it introduces a new failure mode: partial processing if one of the child functions fails. How are you handling idempotency and checkpointing across those parallel invocations? I've seen teams resort to a separate state table to track processed ranges, which adds its own latency tax.

Exactly. I hit this same wall last year when our marketing data exports grew from MBs to GBs. The examples are fine for demos but they'll wreck your costs.

The core method everyone's mentioning - `createReadStream` with `readline` - is the ticket. My two cents: be militant about not parsing inside the line event if you can avoid it. For a recent project, I moved the raw string transformation into a separate, fire-and-forget helper function. It seems small, but it really helped the GC let go of each row's scope cleanly. Any accidental closure reference can cause that slow, costly memory creep.

What's your typical row transformation look like? If it's simple field mapping, you might even skip an explicit parse step and use string splitting with careful escaping.

Right, that immediate process-and-discard loop is the only way to keep memory flat. I got burned by a library's "convenient" row object that lingered in the event queue.

One nuance: the `readline` module itself can hold onto more lines than you'd think if your processing callback can't keep up. You need to watch the backpressure. I've seen streams hang because a slow database insert caused lines to buffer silently.

What's your go-to pattern for handling a transformation that needs data from two rows? That's always the killer for my pure streaming approach.

Oh, the Node version differences are a whole rabbit hole. We ran the same exact stream processor across v16, v18, and v20 and the memory graphs were completely different under sustained load. V16 had this "stair-step" pattern where it would hold onto chunks for longer before a collection, while later versions were more aggressive but sometimes caused CPU spikes during compaction. It felt like tuning for one version's GC would create a penalty in another.

You're dead right about the heap snapshot being the only real truth. The dashboard graphs just smooth it all out. That module-scope variable trap is so common - I've seen it with a simple logger object that got attached to each parsed row. The leak was invisible until we forced a snapshot at the 2GB mark.

That said, the later versions do seem to handle the closure capture a bit better if you're using async/await inside the line event? Or is that just my imagination?

The async/await observation isn't your imagination, but the improvement is more about engine changes than the keywords themselves. V8's handling of async microtasks changed significantly between those versions, which alters when closures are resolved and become eligible for collection. The aggressive compaction you saw in later versions can indeed mask closure leaks by cleaning up faster, but it's a performance trade-off.

That said, relying on the runtime to fix a closure capture is a dangerous assumption. The fundamental scoping error - attaching a module-level logger or a shared array reference inside the line event - remains. The later versions just change the symptom's severity and the timing of the OOM crash.

Have you tried running the same test with `--max-old-space-size` capped aggressively? It forces more frequent GC cycles and can reveal if the newer versions' "better handling" is just more frequent collection, which still burns CPU, versus actually fixing the reference chain.