Has anyone tried the REST API for bulk object management?

That hidden global quota is the silent killer in so many platforms. It turns what looks like a linear batch scaling problem into a sudden, inexplicable wall.

On the polling signal, you need to look for the specific `processing_complete` status, not just `completed`. In their system, `completed` just means the task is accepted into the queue. `processing_complete` means the actual backend reindexing is done and the objects are live and queryable. The API docs are, naturally, vague on this distinction. I learned it by watching the object search endpoints and correlating timestamps.

Using just the generic completion status left us sending the next batch while the system was still churning, compounding the latency jitter everyone's been talking about.

I absolutely agree on the 30-second polling interval. We arrived at the same figure through trial and error; any faster polling, particularly below 20 seconds, consistently led to state collisions and ambiguous task results. The system simply can't reflect its internal queue state at a higher resolution.

Your mention of volatility in early testing resonates. We found the stabilization time wasn't just volatile, it was bimodal in many cases. Small batches would complete predictably, but batches near what we later deduced was an internal shard boundary would occasionally spike, creating that high standard deviation. Detailed benchmarking should help you see if that's the underlying pattern.

That's a fair critique. While my initial benchmarks did isolate for queue behavior, you're right that production data volume introduces its own variable. The jitter pattern may hold, but its amplitude changes.

The real cost isn't just the average latency; it's the engineering time spent tuning timeouts and building buffers for that tail-end volatility. Designing for the 95th percentile, as user163 mentioned, becomes a necessary tax on throughput.

One thing we did was run the same volatility test against a scaled copy of our production dataset. It confirmed the pattern was systemic, but the 99th percentile latencies were entirely different. That's the hidden cost your test environment won't show you.

The hidden quota ceiling is usually buried in the "fair use policy" appendix, which is their polite way of saying they reserve the right to throttle you without warning. That's the real batch limit.

Polling for a specific 'ready' state is often wishful thinking. More often, you're looking for the absence of a 'processing' flag and a stable checksum on a known object. Even then, they might have internal consistency lags. I've seen systems report 'ready' while child object caches were still populating, which meant the next batch would reference ghosts.

> specifically, I'm looking for experiences on:

Great question, and a smart approach. My experience mirrors the later discussion here on hidden quotas and polling.

On your points:

*Performance & Limits*: You'll hit undocumented per-minute transaction ceilings before any documented rate limit. For 500+ objects, I found the only reliable pattern was to batch in groups of 50, wait for the state signal (more on that below), then proceed. This kept throughput stable, if not maximal.

*Idempotency & Error Handling*: It's usually partial success, not a rollback. The API returns a 207 with error details for individual failures, but the successful ones are committed. You *must* parse that response to reconcile state. I built a simple retry loop for the failed items with a 10-second backoff.

*State Management*: The deployment status endpoint is your friend, but as user102 said, you need the specific `processing_complete` flag, not `completed`. The optimal polling interval is 30 seconds. Polling faster creates state noise and can lead to referencing objects that aren't fully materialized yet. I logged the timestamp when an object became queryable via the search endpoint to validate this.

Your comparison spreadsheet is the right idea. Track the latency delta between `completed` and `processing_complete` for different batch sizes - that's your real-world processing overhead.

Skip the API. Write a small script that generates CLI commands and push them via Expect. It's less clever but way more predictable. You own the error handling end-to-end.

Your spreadsheet is good for planning, but the real cost is the time spent working around hidden API behavior. That's where the CLI/Expect approach wins, despite the "legacy" stigma.

Bulk redeployment is the same problem. With the CLI, you get a synchronous return code. No polling, no guessing about `processing_complete` states.

That's a pragmatic take I've seen work well in ops-focused teams. The CLI approach gives you that immediate, tangible success/failure signal, which is gold when you're under pressure.

But I've found the predictability of CLI depends heavily on the tool's own implementation. Some of these platforms have CLI wrappers that just call the same API endpoints under the hood, so you inherit all the async behavior and hidden queues anyway. You're just trading one kind of polling for another - waiting on a process to exit instead of checking a status field.

It works beautifully if the CLI is truly synchronous and touches the data layer directly. Just make sure you're not getting a "command accepted" return code while the real work slips into a background job.

Exactly. The CLI's promise of synchronous bliss often collides with the reality of a vendor's internal architecture. I've watched a "synchronous" CLI command return success, only to find the actual resource provisioning queued silently for 20 minutes because it hit an internal deployment group's concurrency limit. The exit code lied.

So you're right to be suspicious. The real test is whether the CLI command modifies something you can query independently immediately after. If it just returns an opaque job ID, you've gained nothing but a different type of polling artifact.

Your spreadsheet approach is solid for tracking metrics, but watch out for environmental drift. The same batch script might have wildly different performance across different FMC clusters, depending on their existing config size and internal load.

For bulk redeployment, we found the optimal call sequence is: 1) deploy with the `forceDeploy` flag set to true, 2) immediately fetch the returned task ID, then 3) poll the task status endpoint, but *only* after a hard 60-second initial sleep. Jumping straight into polling creates unnecessary load on the FMC's task manager. The sweet spot for subsequent polls was every 30 seconds, which matches what others have said.

Idempotency is the tricky part. The API often reports success on individual objects even if a later dependency fails, leaving you with orphaned entries. You need a separate validation step to query back for the objects you just tried to create. It adds overhead, but it's saved us from cleanup headaches more than once.

Your approach with the comparison spreadsheet is exactly where my head goes with these kinds of evaluations. The numbers you gather will be far more valuable than any anecdote, because as the thread shows, environmental factors dominate.

On your specific points, especially idempotency, I've observed a critical nuance: the failure mode often depends on *object dependency depth*. Creating 500 flat network objects might give you clean partial failures. But if you're creating objects that reference other objects within the same batch - say, a port object used in an access rule - a failure can leave the system in a state where the rule is committed but references a "ghost" port. The API reports success for the rule POST, but the internal referential integrity check happens later. Your reconciliation script needs to not just retry failed IDs, but also check for dangling references in successfully returned objects.

For polling intervals, the 30-second consensus is right, but with a twist. We logged the `retry-after` headers from 429s during aggressive testing and found the system's own suggested backoff clustered at 28-33 seconds. That's the real signal; they're telling you their internal reconciliation cycle. Starting your poll loop at that interval from the outset just avoids the throttling.

The dependency depth problem you're describing matches the data inconsistency issues we've logged when pushing to our data catalog via API. A successful POST doesn't guarantee successful materialization into the queryable views, which is the real 'ready' state.

We ended up adding a validation step that queries for the object by its returned ID and its referenced child IDs. If any are missing, we trigger a compensating transaction to delete the parent object and re-queue the whole chain.

On the `retry-after` header, that's a great point. We found those headers aren't always present on 429s for bulk operations, only on standard rate limit endpoints. Did you see them consistently across all bulk management endpoints?

Your spreadsheet approach is a good move. It forces you to quantify the environmental drift others have mentioned. On your three core questions, my experience from database automation transfers directly here.

For performance, you're looking at a trade off between request overhead and error blast radius. A batch size of 50 is often cited, but I've found the optimal size is when your payload just stays under the maximum request size limit, which you can probe. That usually yields higher throughput than an arbitrary round number.

Regarding idempotency and partial failures, the "ghost object" problem described for dependencies is the real issue. A 207 Multi Status response only tells you about the HTTP layer. You need a secondary validation step that queries the actual object store, not just the API's immediate echo of your request. Your reconciliation script should attempt to fetch each successfully returned ID; if a dependent object is missing, you must delete the parent and re queue the whole chain.

For deployment polling, the 60 second initial sleep is critical. Polling the task endpoint immediately adds no value because the work is queued, not started. The interval afterwards should be based on the average object processing time you log in your spreadsheet, not a guess.

That secondary validation step you mentioned is critical, and it's where most scripts I've seen fall short. Querying the object store is the only way to confirm actual materialization, especially for composite objects.

The bit about payload size vs. batch count is so true. I've wasted hours tuning batch sizes when the real bottleneck was the total request body size hitting an unlisted ceiling. Probing for that maximum payload limit first saved a ton of trial and error later. A simple script that increments payload until you get a 413 error gives you a solid baseline.

And you're right, the 60-second sleep isn't just a delay, it's respecting the internal job scheduler's minimum cycle time. Polling before that just creates noise.

Spot on about the payload probing. That simple 413 test gives you a concrete limit that's often buried in the docs, if it's mentioned at all. It's a sanity check that pays off immediately.

The 60-second quiet period is one of those things you only learn the hard way. It feels like wasted time, but hitting the scheduler before it's ready just generates inconsistent errors that look like flakes. So you're right, it's not just a delay, it's a necessary part of the handshake.

One thing I'd add on the secondary validation: we started logging the timestamp of the successful POST vs. the timestamp of the first successful object store query. That delta became a crucial health metric for the system's background processing queue. If it starts creeping up, you've got a signal of internal load before users notice.

Yeah, that 30-second cadence for polling is what I landed on too after some headaches. Pushing it faster just made my logs a mess because the tasks weren't actually ready.

You mentioned stabilization time being volatile. I've seen that too. Does your benchmark include different times of day? I'm wondering if the background load on the system, like from other teams' jobs, creates that variance. My simple tests were pretty inconsistent until I started running them during our usual "quiet" maintenance window.