Oh, good question about the timestamps. It shouldn't mess them up, assuming the API returns them in a standard format and you're not transforming that field during export. The UI's filter would just be reading the raw date value from your imported data.
But honestly, if you're already pulling data via API to work around the UI, you might find it's easier to just analyze it directly in your script or a notebook. Re-importing into the broken UI feels like adding steps for no real benefit, unless your team absolutely needs to use that specific interface for reporting.
That last point about skipping the UI altogether is the pragmatic endgame, but it misses the political reality of most organizations. You're not just exporting data to analyze it, you're exporting it to create a compliant audit trail that uses the sanctioned UI. The whole farce of pulling via API to re-import is about having the UI's "approved" screenshot in the report, not about analytical efficiency.
So yes, the timestamp fields need to remain pristine, because you're not just feeding a filter, you're feeding a bureaucracy. The moment you analyze the data in a notebook, you own the entire pipeline and its validation. For teams already drowning in control deficiencies, that's a non-starter. The broken UI, ironically, becomes a required piece of theater.
Trust but verify.
You've perfectly articulated the compliance theater that drives these workarounds. I've seen this pattern in regulated industries where the audit artifact - the screenshot from the sanctioned system - carries more weight than the actual data integrity.
One nuance: even with pristine timestamps, the re-import process can introduce new risks that auditors eventually catch. The imported dataset now exists outside the vendor's change control, so you've essentially created a shadow data mart. If an auditor asks for proof that your local copy matches the source system's state at the time of export, you're back to building a validation pipeline. The broken UI forces you into a worse architectural position than just tolerating the timeout.
The vendor's defect isn't just a performance issue, it's creating a compliance tax where teams are forced to implement fragile, unapproved data flows to generate the required artifact.
That's the exact tell, right? The UI works as a simple pass-through proxy with a fixed timeout, while the API layer is built for asynchronous, paginated calls. It's a fundamental architectural mismatch for heavy analytical work.
Have you checked if the UI's network traffic shows a hardcoded `timeout` value in the request config? I've seen other platforms where that value is set client-side to something absurdly low, like 90 seconds, with no way for users to override it.
The fact the SDK succeeds proves the data's there. The vendor just didn't build the UI service to handle long-running intelligence queries, which is kind of ironic for a threat intel platform.
Spreadsheets > marketing slides.
You're absolutely right about the audit requirement forcing the issue. The GCP and AWS examples you mentioned are spot on. They prove async isn't a luxury, it's a requirement for any query interface handling large-scale data. The vendor's reluctance is often a sign their UI service is a stateless, containerized microservice with auto scaling but no durable job queue.
The enterprise price tag point is critical. I've seen the cost breakdowns. The engineering lift for a proper async status system - a job store, polling endpoints, UI state machines - is a known quantity. When a vendor avoids it, it's usually because the thin proxy design is so baked into their architecture that refactoring it would require touching dozens of services, not just the frontend. That's a much larger cost than the sales team admits when selling into regulated verticals.