Skip to content
Notifications
Clear all

Troubleshooting: Prolexic seems to be dropping legitimate API traffic at peak.

2 Posts
2 Users
0 Reactions
4 Views
(@cost_optimizer_88)
Estimable Member
Joined: 3 months ago
Posts: 95
Topic starter   [#17505]

Alright, let's cut through the usual "scale-to-infinity" marketing haze and get into the gritty, expensive reality. I've been running Prolexic in front of a high-volume, transaction-heavy API platform for about eight months. The DDoS protection itself is solidβ€”I'll give them that. No complaints when the bots come knocking.

However, we've now documented three separate incidents in the last quarter where, during our *legitimate* peak traffic periods (think Black Friday-level API calls from verified partners), Prolexic appears to be the choke point. Our origin servers are lounging at 30% CPU, the network path is clear, but the HTTP 5xx errors spike *before* traffic hits our cloud VPC. Akamai's own dashboards show a "request rate" plateau that aligns perfectly with our error curves, while their "mitigated attack" charts are flat. This isn't an attack. This is them throttling *our* money-making traffic.

The setup is textbook:
- API-first infrastructure (Kubernetes, Nginx ingress)
- Prolexic in "forward/reverse proxy" mode, with a CNAME to their edge.
- All standard "recommended" WAF rules enabled, but with learning mode on for our core endpoints.

Before the usual chorus of "check your configs" starts, we've done the homework. Here's the smoking gun from our logging pipeline, comparing a normal peak vs. a "throttled" peak. The pattern is always the same: a rise in legitimate 429s (Too Many Requests) from the Akamai edge, not our origin.

```json
{
"timestamp": "2023-10-27T14:05:32Z",
"client_ip": "",
"status_code": 429,
"request_path": "/api/v2/transaction",
"response_header": {
"x-akamai-user-rate": "1000",
"x-akamai-user-burst": "200",
"x-akamai-user-action": "throttle"
}
}
```

We're paying for the "Business" tier, which supposedly has no "soft" limits. Yet, here we are.

My theory, backed by some painfully expensive experimentation:
* Prolexic's legacy DDoS architecture has implicit rate limits on *individual* CNAMEs/protections, even for clean traffic, which are not clearly documented.
* Their "auto-scaling" for legitimate traffic has a lag time longer than a burst API call spike, causing legitimate requests to be dropped.
* The WAF "learning mode" might be too conservative, interpreting a sudden, valid surge from a few IPs (our API gateways) as potentially malicious and applying a temporary throttle.

What I want to know from this community:
* Has anyone else seen this behavior with API traffic patterns, specifically?
* Is there a magic incantation for the Akamai account team to get them to adjust these hidden thresholds? Our last ticket resulted in a week-long "investigation" ending with "no issues found on our end."
* Are we better off implementing a more aggressive origin pull-back for certain known partner IP ranges, bypassing some of Prolexic's "scrubbing" logic? This feels like it defeats the purpose.
* Most importantly, **what is the actual, measurable request-per-second ceiling people have achieved on a single Prolexic CNAME before seeing these throttles?** The sales docs are, predictably, silent on this.

I'm already modeling the cost of running our own Anycast stack versus this managed service. The math is getting uncomfortably close, especially when you factor in lost revenue from dropped transactions.


pay for what you use, not what you reserve


   
Quote
(@heidir33)
Trusted Member
Joined: 1 week ago
Posts: 39
 

That's a really specific and frustrating scenario you've documented. It sounds like you've already ruled out the obvious infrastructure culprits. When you mention the HTTP 5xx errors spiking before your VPC, are those errors originating from Prolexic itself? I'd be curious if they are 502/503/504s, which might point to a capacity or timeout issue on their reverse proxy layer, versus a blanket 429 from a rate limit you didn't configure.

Having the "mitigated attack" charts stay flat while your request rate plateaus is the most telling part. It suggests this isn't a security policy kicking in, but a throughput limit somewhere in the service chain. I've seen similar behavior with other managed services where the "learning mode" on WAF rules can still introduce latency or connection limits during extreme but legitimate surges, essentially treating the traffic shape itself as suspicious. Have your Akamai contacts acknowledged that plateau in the request rate graphs as an expected behavior?



   
ReplyQuote