Hey everyone, just wanted to share a recent hiccup we encountered during our Orca Security deployment that might help others planning a rollout.
We were onboarding our AWS environment, which has about 50 engineer IAM roles/users, into Orca for cloud security posture management. The initial onboarding scan worked fine, but we started seeing persistent errors in the Orca dashboard related to "API throttling" and "rate limit exceeded" for IAM and CloudTrail calls. This was surprising because our overall AWS API call volume isn't that high.
After digging in with support, we found the root cause: **Orca's scanner, when inventorying IAM entities, makes individual API calls *per IAM entity* for certain checks.** With 50 engineers, each having inline policies, managed policies, and access keys, the scanner essentially triggered a burst of `GetUserPolicy`, `ListAttachedUserPolicies`, and `ListAccessKeys` calls in a short timeframe. This hit our AWS account's IAM request rate limits.
Here’s a snippet of the type of error we saw in the logs:
```
An error occurred (Throttling) when calling the GetUserPolicy operation
(Rate exceeded)
```
**Our quick fix & best practice for others:**
* **Increase AWS Service Quotas:** We immediately requested and were granted an increase for our IAM API rate limits from AWS Support. This is often the fastest path.
* **Stagger Scans:** We worked with Orca to adjust the scan scheduling to be less aggressive during peak hours for our account.
* **Consider IAM Pathing:** For future scale, we're looking at organizing our human users under specific IAM paths. While Orca still needs to scan them, it can help with overall IAM management.
The lesson here is that even a modest number of IAM identities can generate a huge burst of API calls when a tool like Orca performs deep, entity-level inspection. It's a good idea to preemptively check your AWS Service Quotas for IAM, CloudTrail, and Config before deploying any CSPM tool at scale.
Has anyone else run into similar API limit issues with cloud security tools? Curious how you handled it.
security by default
That's a classic issue when security tooling scales. The per-entity API call pattern is common in scanners that need granular policy detail, but it doesn't align with AWS's service-wide rate limiting.
You mentioned a quick fix but didn't include it. For anyone else hitting this, the standard approach is to request a service limit increase for the IAM API through AWS Support. Be specific: reference the exact operations (`GetUserPolicy`, `ListAttachedUserPolicies`, `ListAccessKeys`) and explain it's for a CSPM tool conducting periodic inventories.
A longer-term architectural consideration is whether you need such fine-grained, real-time scanning for all 50 identities on every cycle. Some teams implement a tiered approach where critical IAM entities are scanned at high frequency, while the bulk are inventoried less often. This requires customizing the tool's scan targets if it supports it, which Orca might not. You're then trading off some freshness for API stability.
CPU cycles matter
Good point about requesting specific limit increases. That request should also include `SimulatePrincipalPolicy` if Orca is checking effective permissions, which can be even more expensive.
The tiered scanning approach you mentioned is smart, but I've found most CSPM tools don't support it natively. We had to build a Lambda that tags IAM roles as 'critical' or 'standard' and use Orca's API to schedule targeted scans - it's extra work, but it kept our baseline API calls manageable.
Cloud cost nerd. No, I don't use Reserved Instances.
You're correct that service limit increases are the immediate path, but they're not always granted quickly. In my experience, AWS Support often requires a multi-week history of throttling CloudTrail events to justify the increase. Teams need to plan for that lead time.
While you mentioned a tiered approach, there's a more foundational step: IAM role consolidation. Many organizations with 50 engineers have far more than 50 IAM entities due to legacy roles and per-service patterns. Aggressively merging inline policies into shared managed policies and reducing total principal count directly lowers the scanner's API call footprint before you even adjust scanning frequency.
If Orca doesn't support targeted scanning natively, their API might still allow you to scope an integration to specific AWS accounts. You could isolate high-velocity IAM changes to a sub-account, limiting the main scan's scope.
every dollar counts
Consolidating IAM roles is sound advice in theory, but it's a political minefield. You'll have 50 engineers who all think their unique, bespoke inline policy is critical. Good luck getting that through change control.
Also, scoping by AWS account only helps if your Orca integration is account-level. Ours wasn't. It was an organization-wide role, so it scanned everything or nothing. We had to script a daily IAM entity dump to S3 and point Orca at *that* to avoid the live API calls entirely. Hacky, but it stopped the throttling.
Your point about AWS Support needing weeks of CloudTrail history is painfully true. By the time you get the limit increase, you've already been blocked on your security rollout for a month.
Interesting that the throttling was specifically for `GetUserPolicy` and similar calls. That suggests Orca is trying to fetch the actual policy document for every inline policy, not just the metadata. That's a much heavier operation than just listing what's attached.
Your quick fix mention got cut off, but if it's a service limit increase, you'll want to monitor your IAM TPS (transactions per second) in CloudWatch after it's approved. The new limit might just move the bottleneck, especially if Orca's scanner runs on a tight schedule and tries to inventory all 50 identities concurrently again. Spreading the scan out over an hour or two might be necessary even with higher limits.
Every dollar counts.
That's a crucial detail to share, because those specific API calls are the bottleneck. The quick fix you alluded to is likely the service limit increase, but as others have noted, that can be slow.
You might also check if Orca's scanner can be configured to run its IAM checks on a staggered schedule rather than all at once during its main scan window. Sometimes the default is to run everything concurrently. A 10-minute delay between each user scan could keep you under the threshold while you wait for AWS to approve higher limits.
—Anita
That's the exact pattern we instrumented during our deployment. The burst of `GetUserPolicy` calls is the primary bottleneck, as each one is a synchronous transaction against IAM's low default TPS limits.
The quick fix of a service limit increase is correct, but you need to quantify the required TPS for your support case. You can derive it from Orca's scan window. If it tries to scan 50 entities in 5 minutes, that's a sustained call rate requirement. Calculate it, then request 20-30% above that to account for retries.
A more immediate tactical adjustment is to modify the scanner's concurrency, if Orca's configuration allows it. Reducing concurrent IAM entity scans from, say, 10 down to 2 will flatten the burst profile and often keep you under the default limit while the increase ticket is pending. You'll find this setting in the integration configuration, not the dashboard.
Data first, decisions later.
Love that you included the specific API call example. That `GetUserPolicy` burst is such a classic culprit.
Your quick fix of a service limit increase is definitely step one. Just a heads up from our rollout: submit that ticket to AWS Support *immediately* and include those exact error snippets. They really do need those CloudTrail events as proof.
While you're waiting, see if you can adjust the scanner's concurrency in the Orca settings. Sometimes it's hidden under "scan performance" or "advanced." Dropping it from the default 10 to 3 or 4 made a huge difference for us and kept us scanning, albeit slower, without hitting the wall.
null
That concurrency setting is often locked behind an enterprise plan. They'll suggest the service limit increase first because it doesn't cost them engineering time to implement a feature.
Also, "scanning slower" often means the data is stale by the time the full inventory completes. You're just trading one risk for another.
Trust but verify.