I've been deep in the weeds rebuilding a batch ETL pipeline on AWS, moving from a traditional EC2 + custom scheduler setup to something more managed. The goal is lower ops overhead, but I'm hitting a wall with cost vs. performance.
Glue is fine, but the dev experience feels clunky. I've been testing alternatives and stumbled across **OpenClaw** (the open-source version of ClawData's platform). The promise of a self-hosted, Kubernetes-native batch framework that uses object storage as the primary file system is intriguing, especially for cost control.
My main hesitation is the virtual file system layer (`ofs` they call it). When you're processing terabytes of daily logs with complex joins, the performance of file system operations becomes the bottleneck. I'm skeptical that anything mounted on S3 can match the feel of even an EBS volume for transient shuffle data.
Has anyone run it at scale?
* What's the real-world latency for recursive file ops on the `ofs` compared to, say, an EFS mount?
* Did you have to implement a caching layer (like Alluxio) to get acceptable performance, or does the native integration hold up?
* How painful is the operational overhead compared to a fully-managed service like Glue or EMR?
I'm about to prototype a job using their Terraform provider for the k8s deployment, but I'd love some field notes before I sink days into it. My test job looks something like this in their YAML spec, but I'm worried the `input_path` operations will crawl.
```yaml
job:
name: session_aggregation
steps:
- transform:
input_path: ofs://raw-logs/year=2024/month=08/day=01/*
output_path: ofs://processed/session/date=20240801
script: |
-- complex windowing and aggregation SQL
```
Any insights on whether the I/O abstraction burns more money in compute time than we'd save on managed service premiums would be golden.