How do I extract raw data from Anomali for a custom external report?

Oh man, I feel you on that API documentation maze. I've been trying something similar for a different project. That "export only what the GUI shows" limitation is such a blocker.

Have you looked at the `/api/v1/events` endpoint specifically? I found I could get closer to raw data there by playing with the `fields` parameter in the query to ask for more columns, but you're right, it's still wrapped in layers. I ended up writing a Python script to recursively unpack the nested JSON and flatten it into a proper table. It's messy, but it worked.

You mentioned a direct database query, did you ever find a backdoor, or is that completely locked down in your setup?

Totally, that `/api/v1/events` endpoint is the right path. The `fields` param is key, but I've found you often have to reference the internal field names, not the GUI labels, which means some trial and error.

On the direct database query - it's locked down tight in our cloud-hosted instance, a complete non-starter. The script approach is the only real way. I actually built a small orchestration in Make (used to be Integromat) that polls that endpoint, flattens the nested stuff for detection metadata, and pipes it into BigQuery. The pain point for me was always the pagination and rate limits when you're pulling a large date range.

Did your script handle the different event subtypes cleanly? I kept running into issues where the structure of the JSON payload would shift slightly between, say, a malware event and a phishing event, and my flattening logic would break.

The pain point you've hit is exactly why so many teams end up building a parallel data pipeline. The API's `fields` parameter is your only real leverage, but as others noted, it requires mapping internal field names, often through reverse engineering. To get the asset context you need, you'll likely have to perform a post-API join in your script, as the endpoint won't natively merge in that external database.

For the latency analysis, you'll need to extract both the event creation timestamp and the first action timestamp from the activity log nested object. It's usually under a path like `event.activity_log.entries`. You can write a transform to parse that JSON array for the first status change to 'in progress' or similar.

The biggest operational hurdle is handling historical bulk extracts. The API pagination limit, combined with the server-side timeout for large queries, means you must chunk your requests by short time windows, like 6-hour blocks, to avoid mid-query failures.

Oh, the `fields` parameter. Everyone's favorite game of "guess the internal key." I've had better luck just pulling everything and filtering client-side after the fact. The API overhead for a few extra fields is trivial compared to the developer hours wasted on that reverse engineering scavenger hunt.

And while chunking requests is the accepted workaround, it's a band-aid on a fundamentally broken bulk export mechanism. I've seen more than one script fail silently when the server-side timeout hits a chunk boundary and returns a 200 with partial data. You end up with gaps in your timeline that you only discover weeks later.

Parsing the activity log for that first status change is also a trap. The structure isn't just nested, it's inconsistent. Good luck if your team uses custom statuses or if the log entries aren't chronologically sorted in the payload. I ended up having to sort by timestamp in my transform, which feels like something the API should handle.

Yeah, the "three layers of proprietary nonsense" is spot on. It's a common pattern with security SaaS products - they abstract the underlying data model to the point where it's opaque.

For raw, row-level data, your only real path is the API. Forget direct DB access on a hosted instance. The trick is to use the `fields` parameter with wildcards, like `fields=*`, to get the broadest possible payload, then write a robust parser for the nested structures. You'll absolutely need to join in your asset priority data externally after the pull.

Just be prepared for inconsistent nesting in the activity log where your latency timestamps live. That's where most scripts fail silently.

Wildcards sound good until you pull ten thousand records and half your fields are null objects with a different structure on the next page. The real "robust parser" you need is for handling API version drift when they silently change a nested key. Been there.

Just pull the ID and timestamp first, then fetch each event individually if you need the deep activity log. It's slower, but you won't miss the data when the bulk endpoint gives you a clean-looking 200 with half the nested fields stripped out.

Agreeing that the API is the only path is correct, but recommending a wildcard `fields=*` is a critical operational mistake. The maximum payload size is often hard-capped server-side, and requesting all fields simply guarantees you'll hit that limit with far fewer records per page. This drastically increases the number of API calls and the risk of hitting rate limits or timeouts during a bulk historical pull.

A more effective method is to first fetch the schema or a single complete record to enumerate all possible fields, then explicitly request only the ones you've confirmed are necessary for your analysis. This keeps the per-record payload lean and allows for larger, more efficient page sizes.

Also, the parser problem for the activity log isn't just about inconsistency, it's about type stability. You'll often find a timestamp field that's a string in one record and an integer in another, which will crash a naive parser expecting one or the other.

> the developer hours wasted on that reverse engineering scavenger hunt.

Those developer hours become infrastructure cost if you're pulling everything client-side. A few extra fields isn't trivial when you're pulling millions of events over a year. The data transfer and compute to filter it adds up.

The silent timeout failure on chunked requests is the real killer. You can't trust a 200. I validate every chunk's record count against the `total_count` in the response metadata. If it's off, I back off and retry the chunk. It's slow, but missing data costs more in faulty reports.

And you're right about the chronological sort - the API returns log entries in insertion order, not timestamp order. I had to add a sort step, too.

The subtype inconsistency is a major parser challenge. I handle it by performing a schema-on-read step before flattening. The script examines the first few records of each new subtype in a batch to dynamically build a mapping for that pull. It's extra logic, but it prevents breaks when a new event type appears.

For pagination over large ranges, I've moved to using the event ID as a cursor instead of timestamps, especially if the API's chronological sort is unreliable. This avoids missing records due to insertion order drift.

That's a smart approach to handle schema drift. I've found that same 'schema-on-read' method crucial when pulling data into a data warehouse, because you're right, new event types appear without warning.

Using the event ID as a cursor is a great tip for reliability, though it can make resuming a failed job trickier if you're not tracking the last ID per subtype. Have you run into that?

Yes, the subtype inconsistency is the single biggest failure point in these pipelines. My approach diverges from typical flattening; I use a two-stage transformation. The first stage loads the raw, variable JSON for each event type into a staging table with a generic JSONB column. The second stage applies dedicated, versioned parsing logic per known subtype, triggered by a `event_type` field. This isolates breakage when a new or modified subtype appears - it simply lands in the staging area with a flag for later development, rather than halting the entire pipeline.

Regarding pagination and rate limits, using the event ID as a cursor, as mentioned elsewhere in the thread, is fundamentally more reliable than relying on timestamps for large historical pulls. It eliminates the risk of missing records due to clock skew or insertion order. The trade-off is you lose the ability to filter by date at the API level, requiring a full scan. For ongoing incremental pulls, I combine the two: use a timestamp filter for the initial window, then switch to an ID cursor for robustness after the first page.

The two-stage pipeline with a versioned JSONB staging table is architecturally sound. It's essentially implementing a dead-letter queue pattern for schema evolution. The critical operational detail is monitoring the volume of records flagged for later development, as that queue can become a data landfill if not actively processed.

> use a timestamp filter for the initial window, then switch to an ID cursor

This hybrid approach is pragmatic, but the transition point is tricky. If your timestamp filter for the initial window is too broad, you may still face insertion-order issues at the boundary where you switch to the ID cursor. I've found it necessary to overlap the two methods, fetching a small buffer of records by ID from before the timestamp cutoff to guarantee no gaps.

Your method requires maintaining that versioned parsing logic, which becomes a library of its own. How do you handle the deprecation of old parsing modules when a subtype's structure changes irreversibly?

Direct database access is a non-starter on hosted Anomali. The API is your only viable path. The aggregated endpoints are useless for your use case.

You need the activity log API. That's where your event generation timestamps live. Fetching each event individually is too slow for a dashboard, so you'll need to use pagination with explicit field selection, not wildcards. Sort by insertion ID, not timestamp, to avoid gaps.

Your biggest hurdle will be normalizing the subtype schemas for the latency calculation. Don't try to parse it inline. Dump the raw nested JSON, then apply your logic in a separate step.

Monitoring the dead-letter queue is crucial, totally agree it becomes a data graveyard otherwise. We set up a simple dashboard that tracks the count of unparsed records per subtype and alerts if it spikes. It forces us to actually write the new parsing logic.

> How do you handle the deprecation of old parsing modules when a subtype's structure changes irreversibly?

We version the parsing modules alongside the data itself. The staging table has a `parser_version` field. When a subtype changes, we deploy a new versioned module, and new records get tagged with that version. Old records stay with their original parser version, so historical reports stay consistent. We only backfill if there's a business need.

The overlap technique for the timestamp-to-ID transition is smart. We do something similar, but we also checksum the overlapping records to deduplicate. Adds a bit of processing but saved us from double-counting a few times.