Skip to content
Notifications
Clear all

Help: Can't get the API to reliably pull all incident data.

4 Posts
4 Users
0 Reactions
61 Views
(@charlie99)
Reputable Member
Joined: 4 months ago
Posts: 310
Topic starter   [#26989]

Hey folks, I've been hitting a wall for the last three days and need some collective brainpower. I'm trying to use the Cortex XDR API (specifically the `/incidents/get_incidents/` endpoint) to pull incident data into our internal analytics dashboard, but I'm running into a major consistency issue: **the API isn't reliably returning all incidents, especially when dealing with large date ranges or high volumes.**

My use case is building a near-real-time dashboard of security incidents, mapped to our asset database. The goal is to have a unified view for our SOC and data teams. I'm using a Python-based ETL pipeline that runs every hour, pulling incidents from the last 7 days to ensure we catch any updates to existing incidents.

Here's a simplified version of my current approach:

```python
import requests

def fetch_incidents(from_date, to_date):
url = "https://api-xxxxxxxx.security.paloaltonetworks.com/public_api/v1/incidents/get_incidents/"
headers = {
"Authorization": "Bearer ",
"Content-Type": "application/json"
}
payload = {
"request_data": {
"search_from": from_date,
"search_to": to_date,
"limit": 100
}
}
response = requests.post(url, headers=headers, json=payload)
return response.json()
```

The problem is in the pagination and filtering. Sometimes, especially during busy periods, I'll get a response with `total_count: 150` but only receive, say, 80 incidents across all pages. Other times, incidents that I know exist (because I see them in the XDR UI) are completely absent from the API results.

Things I've already checked/tried:

* **Pagination:** I'm handling the pagination using the `next_page` token. That part seems to work, but the initial `total_count` feels off.
* **Rate Limiting:** I'm staying well below the documented API rate limits, and I'm implementing exponential backoff for 429 errors.
* **Time Zones:** I've quadruple-checked that my timestamps are in UTC and formatted correctly (milliseconds since epoch).
* **Filtering:** I've tried pulling without any extra filters, just the time range, to eliminate any filter-induced loss.

My main theories are:

1. There's some undocumented limit or threshold on the number of incidents returned, even with pagination.
2. The `search_from` / `search_to` parameters behave differently than expected when incidents are updated post-creation.
3. The API might be prioritizing "recently modified" over "created in range," and I need a different query approach.

Has anyone else built a data pipeline off this API and faced similar issues? I'm particularly interested in:

* Any hidden pitfalls with the date range parameters?
* Is there a more reliable way to chunk the requests (e.g., by day, or by incident ID ranges)?
* Should I be using the `/incidents/get_incidents_extra_data/` endpoint differently?

I'm deep in the data integration weeds on this one, and any workflow reports or code snippets you can share would be a lifesaver. My dream is a robust, idempotent data feed into our data lake, but right now the holes in the dataset are a blocker.


Data nerd out


   
Quote
(@ellaq)
Honorable Member
Joined: 4 months ago
Posts: 411
 

Ugh, that endpoint's pagination can be really finicky, especially with time-based queries. You've only shown the `limit` param in your snippet - are you also handling the `next_page` token that gets returned in the response? It's absolutely required to get a complete set.

If you are using it, my bet is on the date filter itself. The API sometimes treats `search_from` and `search_to` more like filters applied *after* the initial pagination logic, not as the primary query scope. I've had way more luck using a different strategy: pulling everything by creation time *without* a date filter, using only pagination, and then applying my own date windowing locally. It's more data to handle upfront, but the results are consistent.

Also, for your hourly pull, are you accounting for incidents that were updated within your 7-day window but created earlier? The `modification_time` field might be a better anchor than `creation_time` if you're trying to catch all changes.


Pipeline is king.


   
ReplyQuote
(@ci_cd_crusader)
Honorable Member
Joined: 5 months ago
Posts: 430
 

Missing the closing bracket on your payload and the actual request call, but I can see the core issue. That `limit` param alone won't get you all the data - you must loop using the pagination token from each response. The API's default limit might be lower than 100, and pagination is the only reliable way to get completeness.

Your hourly incremental pull is also problematic. Relying solely on `search_from`/`search_to` for updates can miss incidents modified within that window but created earlier. You should track the last high-water mark of `modification_time` instead of just creation time. Here's a pattern I've used:

```python
all_incidents = []
next_page = None

while True:
payload = {"request_data": {"limit": 100}}
if next_page:
payload["request_data"]["next_page"] = next_page
# optionally add your date filter here, but beware the caveats user742 mentioned

response = requests.post(url, headers=headers, json=payload).json()
all_incidents.extend(response.get('reply', {}).get('incidents', []))
next_page = response.get('reply', {}).get('next_page')

if not next_page:
break
```

Are you storing the raw incidents somewhere before transformation? If not, you're losing the ability to replay or backfill when the logic changes.


Commit early, deploy often, but always rollback-ready.


   
ReplyQuote
(@data_diver_dan)
Honorable Member
Joined: 7 months ago
Posts: 455
 

You're right to focus on the `limit` parameter being insufficient on its own. However, even with proper pagination, the combination of date filters and high volume can cause race conditions in the data retrieval.

A more deterministic method is to paginate using a strictly increasing field, like `creation_time` or the incident ID itself, as your cursor. Instead of relying on the API's `next_page` token for the entire date range, segment your pull into smaller, non-overlapping windows based on that monotonic field. Process each window with its own pagination loop. This reduces the chance of incidents being skipped due to internal sorting quirks when the result set is large.

I've found that for hourly incremental pulls, you need a stateful checkpoint. Store the maximum `modification_time` from the previous successful run and use that as your `search_from` in the next execution. This captures updates to older incidents, which a pure `creation_time` filter will miss.


Garbage in, garbage out.


   
ReplyQuote