I'm starting to use CloudGen for a data engineering project, moving some ETL pipelines to the cloud. The firewall and access controls seem solid, but the cloud sync feature has been giving me headaches.
I've had sync jobs hang indefinitely without clear errors, and sometimes config changes just don't propagate. My team is considering it for a wider rollout, but this feels like a big risk. Has anyone else run into this? Is there a specific setup step I'm missing, or is this a known issue?
You're definitely not alone, I've seen similar sync weirdness with CloudGen on AWS. The hanging jobs are frustrating because the logs just... stop. For us, it often came down to IAM permission boundaries on the service role that weren't obvious at first. The config not propagating, though, that sounds like the eventual consistency problem we hit if we made changes too quickly back-to-back. We had to build in a mandatory wait between certain operations, which felt like a workaround. Have you checked if your sync target (like S3 or a container registry) has any lifecycle or bucket policies that might be intercepting the process?
cost first, then scale
I've been running CloudGen sync jobs between Azure and GCP buckets for about six months, and your description lines up with my initial experience, particularly the hanging without clear errors. The logs would often just show the sync as "active" in the console while no network traffic was observed.
For me, the fix wasn't just IAM. The sync service has a hidden dependency on the underlying compute instance's metadata service for temporary credentials. If that instance is under heavy load or has a network hiccup, the sync job can enter a zombie state, waiting on a credential refresh that never completes or times out silently. You have to monitor the instance health metrics alongside the job logs.
On the config propagation delay, I replicated that by making a change to a pipeline, then immediately triggering a sync. The sync would often pick up the *previous* state. I wrote a small script to check the config version via their admin API before allowing a sync to start, which solved it. It's a design flaw, I think, in how they snapshot the state at job initiation versus polling for changes during the job runtime.
throughput first