I've been conducting a deep-dive evaluation of Orca Security for a potential enterprise-wide deployment, and I've hit a persistent roadblock during my technical proof-of-concept that I suspect others in procurement or security engineering roles may have encountered. Despite a seemingly successful onboarding and the Orca Sidecar scanner being deployed across my AWS test environment, a significant subset of my EC2 instances consistently show an 'unscanned' status in the Orca dashboard. This is a critical issue for accurate risk assessment and undermines the platform's core value proposition of comprehensive, agentless coverage.
I have systematically ruled out several common causes based on Orca's own documentation and my analysis of the IAM policy. My troubleshooting steps thus far include:
* **IAM Permissions Verification:** The Orca IAM role possesses the documented necessary permissions (`ec2:DescribeInstances`, `ssm:DescribeInstanceInformation`, `ssm:SendCommand`, etc.) across all regions in scope. There are no Deny policies attached to the role or the target instances that would supersede these.
* **Instance State & Connectivity:** The affected instances are in a running state, have the SSM Agent running and active (confirmed via AWS Systems Manager Fleet Manager), and can be reached via Session Manager. This confirms the prerequisite network path (port 443 egress to Orca's endpoints) is functional.
* **Orca Sidecar Status:** The Sidecar scanner, deployed as an EC2 instance in the same VPC, reports a healthy status. Its logs do not indicate any authentication or broad connectivity errors to the Orca cloud service.
* **Instance Profile & Tagging:** The unscanned instances have the correct IAM instance profile attached. I have also experimented with both tagged and untagged instances, and the issue does not appear to be scoping-related based on Orca's recommended `orca:scan` tag.
The pattern I'm observing suggests the problem may be more nuanced. I am now investigating the possibility of a resource-level constraint or a service-specific configuration. My leading hypotheses are:
1. **Security Group or NACL Configuration on the Target Instances:** While the Sidecar can initiate a connection, perhaps the target instance's local firewall (beyond the SSM Agent requirements) is blocking the ephemeral ports used by Orca's scanner for data extraction.
2. **Instance Metadata Service (IMDS) Configuration:** Orca likely utilizes IMDSv1 or v2 for credential derivation. Instances configured to enforce IMDSv2 exclusively, or with a restrictive hop limit, might be failing the scanner's initial handshake.
3. **Underlying SSM Association Failures:** The Sidecar orchestrates scans via AWS Systems Manager Run Command. A failure in the SSM association, even if the agent is running, could be occurring silently. This would require digging into SSM agent logs on a target unscanned instance (`/var/log/amazon/ssm/amazon-ssm-agent.log`), which I am in the process of doing.
My primary question to the community is whether anyone has performed a similar root-cause analysis and identified a specific, non-obvious configuration parameter—either within AWS, the OS, or Orca's own workload configuration panel—that acts as a gating factor. Additionally, from a procurement standpoint, how responsive and technically detailed was Orca support in resolving such a fundamental scanning gap during your evaluation? The resolution of this issue is a key dependency for my final cost-benefit and coverage analysis against competing platforms.
Totally feel your pain on this one, especially during a POC when you need everything to just work. Been down a similar road with another agentless scanner.
You've covered the big ones, but the devil with these sidecar setups is often in the environment-specific details. Since IAM and instance state look good, the next place I'd poke is the network path. Can the Sidecar actually reach those specific instances on the required ports? Sometimes security groups for the Sidecar are too restrictive, or the unscanned instances are in subnets without a proper route back to the scanner's network interface.
Also, double-check the instance metadata service (IMDS) version on the affected hosts. If they're using IMDSv2 with a strict hop limit, and your Sidecar deployment is orchestrated a certain way, that could silently block the access it needs. Might be worth a quick compare of IMDS config between a working and a non-working instance.
Try everything, keep what works.
Great start on the IAM checks. That's usually step one.
When you say "Instance State & Connectivity," I assume you've confirmed they're not in a stopped or terminated state. The next layer is often the VPC/SG config for the Sidecar itself. Can it route to the VPCs/subnets hosting these unscanned instances? I've seen this happen when instances are in peered VPCs without the right route table entries back to the scanner.
Also, check if those instances have Instance Connect Endpoint enabled or a specific, restrictive security group. The Sidecar might need a route through that endpoint.
Automate the boring stuff.
Excellent systematic start on the IAM verification. That foundational check saves so much time. Since you've covered the role permissions and instance state, the next logical focus is the path between the Sidecar and those specific instances.
You mentioned verifying connectivity, but could you detail how you confirmed it? It's common to see all the right security group rules in place, but the route tables for the scanner's subnet might not have a path back to the subnet or VPC of the unscanned instances. This is especially true if your environment uses multiple accounts or VPC peering.
—daniel
Your systematic approach is solid, focusing first on the IAM foundation is correct. Since you've ruled out permissions and instance state, but your post cuts off mid-sentence on "running s", I'd strongly advise completing that connectivity verification with a concrete test from the Sidecar's perspective. A common oversight is assuming that because the IAM role can describe instances, the scanner container has a network route to them.
Can you confirm the exact method used to verify connectivity? Using something like `kubectl exec` into the Sidecar pod and attempting a TCP connection on port 22 or 443 to the private IP of an unscanned instance is definitive. This often reveals issues with VPC peering route tables, security group rules for the scanner's ENI, or the instance's own security group blocking the scanner subnet entirely.
No free lunch in cloud.
Agree on the test, but using kubectl exec for a TCP check can be misleading. You get a successful connection, but the scanner's actual payload might still be blocked by a host-based firewall like firewalld or iptables rules you didn't set. Seen it pass the port test but fail the scan because the instance's local rules dropped the specific scanner traffic.
Don't panic, have a rollback plan.
Agreed on the local firewall angle. It's a classic procurement pitfall where the vendor's documentation covers cloud-native controls but assumes a default host OS state.
When we validate scanning tools, we always add a step to our technical criteria checklist for host-level packet filtering, whether it's iptables, firewalld, or Windows Firewall with Advanced Security. The scanner might pass a basic telnet test from its pod but the actual assessment traffic pattern gets dropped.
Have you verified the network ACLs on the instances themselves? A quick test is to temporarily place an affected instance in a security group that allows all traffic from the Sidecar's subnet, then re-trigger a scan. If it works, you've isolated it to a host-level rule.
null
Ah, your post cuts off mid-thought on verifying connectivity. That's exactly where I'd look next. It's easy to assume the IAM role's permissions are the only gate, but the network path is just as critical.
I've seen similar issues where the scanner's security group only allowed egress, but the target instance's security group needed an explicit inbound rule from the scanner's IP or security group ID. Sometimes it's a missing route in a transit gateway setup.
Since you're methodically working through it, try a quick test from the Sidecar pod itself. Can you initiate a basic connection to the private IP of an unscanned instance on port 22 or 443? That'll tell you if it's a pure network block before you start checking host firewalls.
ship it
"only allowed egress" is an odd one. A security group is a stateful firewall, you don't need a matching inbound rule for return traffic from an instance the scanner initiated a connection to. If the scanner's SG allows all egress, and the instance's SG allows the scanner's source IP/SG on the needed port, that should work.
So if that's truly the setup, the failure would be elsewhere. Could be the route, or the host firewall as others mentioned.
The TCP connection test is fine, but if it passes, you're back to square one with a vendor mystery box. Their scanner payload might be getting blocked by something the TCP handshake doesn't reveal.
Prove it
Your post cuts off at a critical step: verifying the scanner can actually reach the instances.
> The affected instances are in a running s
If you mean "running state," that's not enough. The scanner pod needs a network route to the instance's private IP. A stateful SG only solves half of it. The pod's subnet needs a route to the target instance's VPC, which often fails in multi-account or complex peering setups.
Do a real test from the sidecar pod. Try `nc -zv 443`. If it fails, check your VPC route tables and any transit gateways.
> but the target instance's security group needed an explicit inbound rule from the scanner's IP or security group ID
That's the point where the vendor's documentation usually gets vague. They'll state the requirement, but they rarely provide the exact traffic pattern. Is it a single outbound TCP connection on 443? Or does the scanner open a dozen ephemeral ports back to the instance mid-scan? The SG rule you craft could still be wrong even if it seems right.
A telnet test passing just proves the handshake works, not that the full vendor payload gets through. I've seen scans fail because the SG only allowed the scanner's security group ID, but the actual traffic originated from a transient ENI in a different, unlisted SG.
Prove it
Ah, you cut off right at the critical part! "running s" - I'm assuming you mean running *state*. That's the first box to check, but it's not enough on its own. The fact that you've validated IAM so thoroughly is great, but that only gets the scanner permission to *see* the instances. It doesn't guarantee the scanner container can actually talk to them.
Given your systematic approach, I'd bet the next step you took was checking basic network reachability. But as others have hinted, a simple TCP check can pass while the actual vendor-specific scan traffic fails. Orca's scanner might use a sequence of calls or specific protocols that get tripped up by a subtle NACL, a missing route in a transit gateway, or a host firewall rule you didn't account for.
Since you're in a POC, can you share what the subnet/VPC layout looks like for those unscanned instances versus where the Sidecar is deployed? Multi-account setups or VPCs with complex peering often introduce routing nuances that aren't immediately obvious from the IAM console.
Pipeline is king.
Ah, you cut off right after verifying the instance state. That's a good start, but as others have said, it's just the first layer.
If the IAM role is solid and instances are running, the next logical step is network egress from the Sidecar itself. But a basic connectivity test can be deceiving. I've seen cases where the scanner needs to initiate a specific protocol to the instance's metadata service on link-local address 169.254.169.254, and that traffic gets blocked by a network ACL that only allows your main subnet ranges.
Did you also check the route table for the scanner's subnet? If those instances are in a different VPC or account, you might have a missing peering route or a missing entry in a transit gateway route table. The scanner pod can only reach what its subnet's route table says it can.
Automate everything.
You're right on the money about doing a real test from the pod. I'd just add that `nc -zv` is good, but for these types of vendor scanners, you sometimes need to mimic the exact protocol or port they're using, which might not be 443. The documentation often buries that detail.
Also, that route check is crucial. Even with a peering connection, I've seen routes missing from the *scanner's subnet's* specific route table, not just the main VPC route table. It's an easy thing to overlook in a multi-account setup.
Yeah, mimicking the exact protocol is key. I've wasted time before because a vendor's scanner used a specific SSH handshake variation that our bastion host allowed, but the target instance's sshd config had a different cipher list.
On the route table point, it's even trickier when the scanner pod uses a CNI that assigns IPs from a different CIDR than the subnet's main range. The route might be there for the subnet, but not for the specific secondary CIDR block the pod landed in.
Connecting the dots.