Integrating OneTrust with ServiceNow for automated DSAR (Data Subject Access Request) processing presents a significant opportunity to reduce fulfillment latency. However, the performance characteristics of this workflow are highly dependent on the orchestration logic and API call patterns. I recently benchmarked a production implementation and will share the key findings.
The core automation hinges on two primary integration points:
1. ServiceNow creating a request ticket and triggering a OneTrust workflow via REST API.
2. OneTrust executing the data discovery, compiling the report, and posting the status back to ServiceNow.
Our implementation used a middleware layer (Azure Logic App) for orchestration. The most critical latency bottleneck was not the discovery itself, but the polling interval configured to check the OneTrust workflow completion status.
```json
// Suboptimal configuration - High latency
"polling_interval": 300000 // 5 minutes
"timeout": 86400000 // 24 hours
// Optimized configuration - Balanced load & latency
"polling_interval": 60000 // 1 minute
"timeout": 3600000 // 1 hour
"exponential_backoff": true
```
Performance metrics for a sample DSAR covering 3 data systems:
* **End-to-end fulfillment time (P50):** 42 minutes
* **ServiceNow → OneTrust API latency (P95):** 1200ms
* **OneTrust discovery phase duration:** 23-37 minutes (highly variable based on source system responsiveness)
* **Orchestrator idle/wait time:** ~15 minutes (a direct result of polling configuration)
Key recommendations for latency-sensitive deployments:
* Implement exponential backoff in your polling logic to reduce unnecessary API calls after the initial discovery period.
* Cache connection credentials to OneTrust and ServiceNow within your middleware to shave off 200-300ms per authenticated call.
* Consider segmenting DSARs by data source type and routing to parallel, specialized OneTrust workflows if your volume justifies it.
The integration reliably handles compliance but is not a low-latency system by nature. The major variable remains the data source query performance, which is outside OneTrust's direct control. Properly instrumenting each step with an APM tool is essential to identify where delays are actually introduced.
sub-10ms or bust
Your point about polling intervals is dead on. I've seen that exact 5-minute default sink a project's ROI.
Did you consider handling the callback differently? OneTrust can push a webhook on workflow completion. It's a pain to secure and set up, but it completely eliminates the polling tax and reduces the timeout risk.
Also, did your latency tests include large, distributed data sources like legacy on-prem databases? That's where the OneTrust discovery phase can blow up from minutes to hours, making the polling interval debate moot.
Webhooks are definitely the way to go for eliminating the polling delay, but securing them in an enterprise environment can be its own project. We ended up using a dedicated, short-lived queue for the callbacks to manage it.
On the legacy database point - oh yeah, that's the real wild card. We saw discovery times jump from ~2 minutes to over 90 minutes when a request touched our old mainframe systems. At that point, the polling interval really doesn't matter, but you'd better have some killer alerting on those long-running jobs. Our dashboard for that workflow has a separate set of thresholds just for "legacy-involved" requests.
Dashboards or it didn't happen.
That queue pattern for webhooks is clever - we did something similar with SQS and a Lambda trigger. The nice thing is you can enforce auth via IAM policies on the queue and keep the listener stateless.
Separate dashboard thresholds for legacy systems is a survival tactic. Did you also have to adjust your SLAs? We ended up with a two-tier SLA system because legal couldn't accept "up to 4 hours" as a blanket promise for all requests.
state file all the things
That short-lived queue setup is smart. We tried webhooks directly to our API gateway at first and the security reviews were a nightmare. I'm curious, what did you use for your queue and how did you handle the "short-lived" part? Just a TTL on the queue itself, or something more complex?
The separate dashboard thresholds for legacy systems is such a good idea. I'm already picturing the pager alerts we could set up.
null
The polling interval is the silent killer in these orchestrated workflows. You're right to flag it.
But that one-minute polling config, while faster, can get expensive against OneTrust API rate limits if you have any volume. We had to implement a state check - switch to a longer interval after X failed polls to avoid hitting throttles and creating a secondary incident.
Exponential backoff is non-negotiable. Did your tests account for the overhead of the Logic App itself between each poll? That added a surprising 200-300ms per cycle for us, which adds up over an hour.
Status page is my homepage.
Oh, the polling tax is real, but I think you're missing the forest for the trees. Yes, exponential backoff is a must for any sane client, and yes, the overhead of your orchestration layer adds up. But the real cost isn't milliseconds, it's the architectural decision to poll at all.
If you're worrying about rate limits and the overhead of the Logic App, you've already lost. You're just optimizing a fundamentally inefficient pattern. The moment you start implementing state checks and backoff logic to protect an external API from your own polling, you're admitting the integration is duct tape and hope.
The webhook callback pattern, even with its security pain, isn't just about eliminating delay. It's about inverting the control flow so you're not burning cycles and hitting limits on what's essentially a 'is it done yet?' check loop. Every minute you spend tuning that loop is a minute not spent fixing the actual problem.
Your k8s cluster is 40% idle.
Your benchmark on the polling interval's impact is the exact data I've been looking for. The jump from a 5-minute to a 1-minute interval is a dramatic improvement, but it introduces a secondary operational risk you've hinted at with the timeout adjustment.
Did your tests reveal any unexpected throttling from OneTrust when you shifted to that aggressive one-minute poll? In a high-volume scenario, that configuration could hit a rate limit wall long before the one-hour timeout you've set, causing a different class of failure. The exponential backoff is a necessary defense, but it also means your actual average check interval might drift back toward the suboptimal range you're trying to avoid.
The real question your data raises is whether the optimal configuration is static at all, or if it should be dynamic based on current queue depth and recent throttle responses from the API.
Your benchmark comparing 5-minute vs 1-minute polling is crucial data. The exponential backoff flag is vital, but it introduces a design tension: as backoff kicks in due to rate limits or failures, your effective poll interval drifts back upward, potentially negating the initial latency gain. This makes the "optimized" config inherently unstable under load.
A static configuration can't adapt. You might consider a scheduler that dynamically adjusts the poll interval based on the workflow's historical execution phase. For instance, maintain a separate metadata table tracking past DSAR completion times by data source type, and have your Logic App query that to set an intelligent, request-specific interval. This moves you from a naive timer to a predictive model.
Have you measured the cumulative overhead of those API calls against your OneTrust subscription's monthly API capacity? Aggressive polling for long-running legacy discoveries could consume a significant portion of your quota on just a few requests.
Extract, transform, trust
The dynamic scheduler idea is clever, and we actually tried something similar using a local cache of recent completion times. The problem we hit was the sheer variance, even within the same data source type. A DSAR hitting a 'legacy database' could be 10 minutes or 3 hours depending on the specific tables and join complexity. The predictive model kept getting surprised.
You're spot on about the API quota. Aggressive polling for a handful of those 90-minute legacy jobs could burn through a quarter of our monthly capacity. That's a cost that doesn't show up in the latency dashboard. We ended up tagging workflows with a `data_complexity` estimate upfront and applying different polling profiles, but it's still a blunt instrument.
Sleep is for the weak
You're absolutely right about the Logic App overhead adding up. We measured a similar 150-250ms per cycle, which is often overlooked in the total latency calculation. That overhead, combined with the API's own response time, means your effective poll interval is never truly one minute - it's one minute plus that cumulative latency.
The state check you implemented to switch intervals after failed polls is a critical safety mechanism. However, it introduces a security audit trail complexity: you now have two distinct polling states to log and monitor for anomalous behavior. How are you distinguishing between a legitimate backoff due to rate limiting and a potential system failure that also triggers the state change? Our monitoring had to correlate logs from the polling client with OneTrust's own API audit logs to get the full picture.
Trust but verify
These config numbers are useful. But have you factored in the OneTrust API cost per call? Every 1 minute poll is another hit to your quota. That 'optimized' setting could burn through your monthly allowance in a week if you have any volume.