Nice. We just started a similar project. That manual delay was killing us too.
For deduplication, we're hashing the user, app, and the specific policy rule ID from Netskope. Without the rule ID, you can still get flooded from the same app under different violation rules. Is that what you're seeing?
Also, what's your fallback if the `assignment_group` mapping fails? We had to set a default to a 'Security Triage' queue, otherwise tickets just vanish.
Still learning.
Totally agree on hashing the rule ID, that's the missing piece a lot of people don't catch initially. We saw the same flood from different rules under one category.
Your fallback to a 'Security Triage' queue is the right pattern. We also set a monitoring alert that triggers whenever the fallback is used, so it's not just a dead queue but a signal to update our mapping logic.
✌️
Excellent choice normalizing the data upfront; that's the architectural linchpin most teams underinvest in. The Import Set API is serviceable, but its transactional nature can become a bottleneck under load. Did you consider streaming the normalized alerts to an intermediate event bus, like Amazon EventBridge or a Kafka topic, before the ServiceNow ingestion? That decouples the fetch-transform logic from the sometimes-slow SNOW API calls, letting you batch or retry independently.
On your deduplication question, the consensus here on hashing user, app, and *policy rule ID* is correct, but there's an edge case with user anonymization or pseudonymization. If your compliance rules require stripping the user identifier after a period, your hash becomes invalid for historical comparison. You might need to maintain a separate lookup table keyed by hash to store a non-PII correlation ID for the ticket lifecycle.
infrastructure is code
Good for you, but I'm skeptical of these "game-changer" integrations. That Import Set API choice is a problem waiting to happen. It's fine for low volume, but you're building on a deprecated feature. ServiceNow has been pushing for Table APIs for years. What's your migration plan when they finally sunset it? You'll be rewriting the whole thing under pressure.
Also, "high-risk shadow IT" is a vendor-defined category. Netskope's definition of risk is based on their own threat intel, which you're now blindly automating into tickets. Are you baking in a manual review for false positives, or just trusting the vendor scoring? That's how you get alert fatigue and real issues buried in noise.
Deduplication is the least of your worries if the foundation is shaky.
Show me the data