Hey folks, diving into a topic I haven't seen much about here: automating data extraction from Akamai Prolexic for internal reporting.
We were manually pulling CSV reports for monthly reviews—super tedious. Their API is decent, but we wanted to automate cost allocation and attack severity trends into our Looker dashboards. Ended up writing a Python script that fetches the data, transforms it, and loads it into our warehouse (Snowflake). Sharing the core part in case it helps anyone.
The main challenge was handling their pagination and date ranges for the "Attack Summaries" data. Here's the key function we use:
```python
import requests
import pandas as pd
from datetime import datetime, timedelta
def fetch_prolexic_attack_summaries(api_key, start_date, end_date):
"""
Fetches attack summary data from Prolexic API.
Returns a pandas DataFrame.
"""
headers = {'Authorization': f'Bearer {api_key}'}
base_url = "https://api.akamai.com/prolexic/v1/attack-summaries"
all_records = []
page = 1
while True:
params = {
'startDate': start_date,
'endDate': end_date,
'page': page,
'size': 100 # max per page
}
response = requests.get(base_url, headers=headers, params=params)
response.raise_for_status()
data = response.json()
attacks = data.get('attacks', [])
if not attacks:
break
all_records.extend(attacks)
page += 1
df = pd.DataFrame(all_records)
# Normalize nested fields like 'attackMetrics'
df = pd.json_normalize(df.to_dict('records'))
return df
```
We then run this monthly, calculate derived fields (like estimated cost impact based on attack duration/type), and push it to Snowflake via a dbt model. Some gotchas we hit:
* The API sometimes throttles under high load—needed to add retry logic.
* Nested JSON for attack metrics requires careful flattening.
* Their date-time formats are UTC but come as strings without timezone info.
This flow lets us correlate attack data with our own billing exports. Curious if others have built similar pipelines? Especially around:
- Mapping Akamai's attack classifications to our internal severity tiers.
- Automating the retrieval of configuration change logs.
Would love to see other approaches or scripts people are using.
--diver
Data is the new oil - but it's usually crude.
Pushing API keys directly into a script like that is a compliance nightmare. You're one config file mistake away from leaking it. That bearer token belongs in a secure secret manager, not hardcoded or passed as a plain function argument.
And where's the error handling? A while True loop with a network call can hang indefinitely. You need timeouts and retry logic, otherwise this fails silently and your dashboard shows stale data for a month.
Also, feeding raw API data straight into pandas and then to a warehouse skips validation. What if the API changes a field type or adds a new nested object? Your transform breaks and you're loading garbage. You need a schema check before the load stage.
— geo