Yesterday at 14:23 UTC, our primary customer-facing API cluster was targeted by a volumetric DDoS attack, peaking at approximately 85 Gbps and 12 million packets per second. We have Akamai Prolexic in front of our infrastructure, and from a purely technical and operational standpoint, it performed its primary function: our origin servers never saw the attack traffic, and our API remained available to legitimate users. The mitigation was fully automated and triggered within 90 seconds of the traffic exceeding our defined thresholds.
However, the event exposed a critical failure in our internal communication and visibility workflows. While Prolexic stopped the traffic, the information flow to our internal teams was chaotic and insufficient for rapid, informed response. I've documented the timeline and gaps below, as I believe this is where many organizations will stumble after assuming the procurement of a solution like Prolexic is the finish line.
**The Communication Breakdown:**
* **Alert Overload & Lack of Synthesis:** Our monitoring systems (internal and Akamai's own alerts) fired independently. We received:
* Akamai Prolexic Kona Site Defender alerts via email (multiple).
* Akamai Edge DNS alerts via a separate portal.
* Internal network flow alerts from our backbone provider.
* Application-level latency alerts from our APM tool.
This resulted in 20+ separate high-severity alerts across 5 different channels (Slack, PagerDuty, email) in the first 3 minutes, with no clear "single pane of glass."
* **Delayed & Manual Data Aggregation:** The Security Operations Center declared an incident but spent the first 15 minutes manually correlating data. Key questions they couldn't answer quickly from the Prolexic portal alone:
* What was the precise attack vector? (We later learned it was a CLDAP reflection attack, but that wasn't immediately clear).
* Were any of our own IPs inadvertently exposed and being targeted directly, bypassing Prolexic?
* What was the geographical source of the attack traffic post-mitigation?
* Was there any collateral impact on legitimate traffic due to mitigation rules?
* **Lack of Integrated Dashboards:** Our internal status page required manual updates. The engineering team responsible for the API had no real-time view into Prolexic's mitigation actions. They were relying on SOC for second-hand information, which created a bottleneck.
**Proposed Remediation & Benchmarking Setup:**
We are now building an integrated dashboard and automating data extraction from Prolexic's APIs to prevent this. I'm also designing a controlled test to benchmark our response workflow, not just Prolexic's mitigation. The goal is to measure "Time to Context" (TTC) in addition to "Time to Mitigate" (TTM).
```python
# Example of a script to poll Prolexic APIs during an incident for key data.
import requests
import json
AKAMAI_API_BASE = "https://api.akamai.com"
ATTACK_ID = "abc123" # Example from alert
def get_mitigation_summary(attack_id):
# This would call the relevant Prolexic API endpoint
# to fetch vector, top sources, impacted ports/protocols.
headers = {'Authorization': 'Bearer ...'}
response = requests.get(f"{AKAMAI_API_BASE}/prolexic/v1/attacks/{attack_id}/summary",
headers=headers)
return response.json()
# This data should be auto-injected into our incident management platform (e.g., Incident.io, Jira).
```
My key takeaway: Prolexic worked as advertised on the network layer, but our internal processes were not "integrated" with it. The post-mitigation phase was more disruptive than it needed to be due to information scarcity. I'm interested to hear from other teams on how you've structured your comms and data flows between Prolexic, your SOC, and your engineering teams. What tools or home-grown dashboards have you built to get a unified view during an attack?
Oh wow, that alert overload part hits home. We use a different provider, but it's the same chaos. You get an email, then a Slack alert from another system, then a PagerDuty page... all saying the same scary thing in slightly different words. You spend the first five minutes just figuring out which one to trust.
The tool did its job, but now your team is scrambling anyway. It feels like winning the battle but losing your mind. How did you guys finally get a single source of truth? Was it someone just taking charge in a chat and saying "ignore everything, look here"?