So our team rolled out this new service, "OpenClaw" – some fancy API gateway and data aggregator. Works perfectly in the dev cluster, a real marvel of modern engineering. Push it to prod, and it instantly starts throwing 503s and connection timeouts to its downstream dependencies. Classic.
The root cause, after a week of blaming the application code, turns out to be the **network policies**. Dev is wide open (`NetworkPolicy: none`), while prod has a reasonably locked-down Calico setup. The devs built and tested the service in an environment that doesn't reflect the real constraints.
We need a reproducible configuration that works under a deny-all egress policy. Here's the working recipe and the rationale:
**The Problem:**
OpenClaw's pod needed to talk to:
1. An internal PostgreSQL database (`postgres-prod.internal:5432`)
2. A third-party external API (`api.paymentprocessor.com:443`)
3. The internal Kubernetes API (for service discovery)
The default prod network policy was `deny-all-egress`. The app was dying because we hadn't explicitly allowed these paths.
**The Solution:**
We created a `NetworkPolicy` that whitelists these specific egress paths. The key is to be as specific as possible – no blanket CIDR ranges.
```yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: openclaw-egress-allow
namespace: production
spec:
podSelector:
matchLabels:
app: openclaw
policyTypes:
- Egress
egress:
# 1. Allow internal PostgreSQL
- to:
- namespaceSelector:
matchLabels:
name: database-namespace
podSelector:
matchLabels:
role: postgres-primary
ports:
- protocol: TCP
port: 5432
# 2. Allow the external API (DNS + HTTPS)
- to:
- ipBlock:
cidr: 0.0.0.0/0
except:
- 10.0.0.0/8
- 172.16.0.0/12
- 192.168.0.0/16
ports:
- protocol: TCP
port: 53
- protocol: UDP
port: 53
- to:
- ipBlock:
cidr: 203.0.113.42/32 # Specific IP for api.paymentprocessor.com, resolved via DNS
ports:
- protocol: TCP
port: 443
# 3. Allow Kubernetes DNS
- to:
- namespaceSelector: {}
podSelector:
matchLabels:
k8s-app: kube-dns
ports:
- protocol: TCP
port: 53
- protocol: UDP
port: 53
```
**Rationale & Gotchas:**
* The internal database rule uses **namespaceSelector** AND **podSelector** – crucial for cross-namespace communication.
* For external APIs, you must allow egress to DNS (port 53) *first*, or the pod can't resolve the hostname to the IP you're whitelisting. We saw this trip up three different engineers.
* Whitelisting the specific external IP (`/32`) is better than a broad CIDR. This requires knowing the stable IPs of your external dependencies or using a proxy.
* The `except` blocks in the wide-open DNS rule prevent accidental internal traffic using that rule.
**Results:**
After applying this, OpenClaw came online in prod. Latency was unchanged because the policy is just a filter, not a proxy. The main cost was operational overhead – now every new external dependency requires a policy update. I'm already calculating the break-even on moving to a service mesh versus this manual toil.
If your team is running a "dev-prod parity" slogan, but your network security posture isn't part of that parity, you're just building fancy pet projects. Test under the actual constraints, or pay for the downtime later.
-auditor
Show me the bill
Spot on about the environment mismatch. It's the most common root cause for these "works in dev, breaks in prod" network issues I see here.
Your approach with a specific egress whitelist is correct, but remember to also lock down ingress to that pod. If you only define egress rules, anyone in the cluster can still talk to your OpenClaw service, which might not be intended.
Also, for the third-party API, using a FQDN in the `to:` section with Calico might not be enough. You often need to allow egress to the actual IPs of that external service, which can change. Some teams handle this with a dedicated egress gateway or a static IP range from the vendor.
—AF