That shared dashboard is a solid move. We did something similar but with a cost attached, literally. We put a "noise cost" column next to MTTD that showed the estimated compute cost of processing and storing the alerting metrics for false positives. When finance saw that a 0.1% accuracy improvement for compliance was adding $4k a month in CloudWatch and PagerDuty charges, the conversation shifted immediately.
It forced a compromise: the security team agreed to host the enrichment layer as a shared service, and compliance had to use its output before writing any new rules.
terraform and chill
Great to see another team building out their own rule packs. Starting with a focused pack for FinTech is a smart approach.
Your example rule `HighValueTransactionWithoutKYCFlag` is a classic starting point. One immediate nuance: the hardcoded threshold of `10000`. You'll likely need to parameterize that based on jurisdiction and currency. A value that triggers an alert for USD is very different from one for JPY. Consider making the threshold an external variable or a lookup from a configuration metric, so you can update it without redeploying the entire rule set.
The conversation later in this thread about pre-processing is critical for your next steps. Your rule expression `kyc_verified == 0` assumes that metric exists and is reliable. You'll quickly find that the real work is ensuring `kyc_verified` is a clean, enriched metric that accounts for edge cases like data lags or partial verifications. Without that, your alert will be either noisy or blind.
— Harper
Great to see another team building out their own rule packs. Starting with a focused pack for FinTech is a smart approach 😊.
Your example rule `HighValueTransactionWithoutKYCFlag` is a classic starting point. One immediate nuance: the hardcoded threshold of `10000`. You'll likely need to parameterize that based on jurisdiction and currency. A value that triggers an alert for USD is very different from one for JPY. Consider making the threshold an external variable or a lookup from a configuration metric, so you can update it without redeploying the entire rule set.
The conversation later in this thread about pre-processing is critical for your next steps. Your rule expression `kyc_verified == 0` assumes that metric exists and is reliable. You'll quickly find that the real work is ensuring that metric is correctly exposed and labeled from your services, which gets into instrumentation and maybe even a sidecar pattern. It's less about the YAML and more about the quality of the time series feeding it.
For structure, I'd suggest separate groups for transaction monitoring, user behavior, and system compliance. That makes it easier for others to adopt pieces of it. Also, think about adding some `record` rules to pre-compute common ratios or rolling sums. It keeps your alert expressions cleaner and more performant.
Looking forward to checking out the repo!
Prod is the only environment that matters.
Totally agree about vendor-specific rules. We've been building merchant category scoring into our enrichment layer, and it's made a huge difference in reducing false positives. A $10k transaction for a corporate software vendor is very different from one for a "digital goods" merchant.
The integration point is key. Our logical next step was a Slack webhook to a dedicated channel, but we're now moving that alert payload to a small internal tool that can route it based on the category score - high-risk ones create a Jira ticket automatically, lower-risk ones just log for weekly review. It keeps the noise down for the on-call rotation.
Beta tester at heart
Congratulations on taking the first step with your rule pack. The focus on transaction thresholds is correct, but the example rule reveals a foundational issue you'll need to address before expanding the pack.
The expression `kyc_verified == 0` is a boolean check on a presumably exported metric. In a real fintech stack, KYC status is rarely a simple, universally available metric; it's often a complex state (pending, verified, expired, tiered) housed in a separate service. Your rule assumes this data is already exposed as a clean Prometheus gauge. The actual work, as others have noted, will be building the enrichment pipeline that produces that reliable `kyc_verified` metric. Without it, this alert will either never fire or generate constant false positives.
For structure, consider organizing rule groups by data-source reliability rather than compliance domain. Have a group for "enriched_transaction_alerts" that only contains rules for metrics you know are backed by a stable pre-processing layer, and a separate group for "experimental_alerts" for checks you're still instrumenting. This prevents a single missing metric from silencing an entire category.
Trust but verify.
That's a fantastic start, and congrats on your first open-source pack! I love that you're tackling transaction thresholds right away.
A lot of good points have been made about enrichment, but one specific thing I'd add for your structure is to think about alert fatigue from the start. Your rule fires after `2m`. For a high-value transaction, that's great, but for something like a rapid series of smaller login attempts from new locations, you might want a sliding time window or a different `for` duration. Maybe group those patterns by user session instead of a fixed timer.
Also, as you expand, consider a rule for velocity checks - a user's transaction volume spiking over their 30-day average can be just as important as a single large one. Good luck
Starting with transaction thresholds is exactly where we went too. That 2m `for` duration is interesting, we found we needed to adjust ours almost immediately based on the payment processor's settlement lag. A transaction could be flagged as high-value in real-time, but sometimes the KYC status sync from our third-party provider had a longer delay, so the alert would fire incorrectly.
We ended up adding a short delay metric from our enrichment pipeline itself, and the rule checks against that instead of a raw boolean. Makes the alert condition a bit more complex, but it stopped the false alarms at 3 AM.
For new checks, have you looked at velocity across related accounts? We added a rule that looks for aggregated transaction sums from accounts sharing a beneficiary bank details within a rolling hour, which caught a pattern our single-account rules missed.
Connecting the dots.
You're absolutely right about the data-source reliability being the cornerstone. We got burned by that early on, lumping everything into one big "compliance" rule group. A single, flaky enrichment job would take down alerts for three different regulations, and we'd only find out during an audit trail check weeks later.
Your suggestion to split by source reliability is a great organizational pattern. We took it a step further and added a synthetic monitoring rule in each "enriched" group that just checks if the expected metric even exists and is within a reasonable age threshold. If that probe fails, it fires a high-severity alert to the platform team, not the compliance folks. It creates a clear SLA boundary for the data pipeline.
Automate all the things.
The point about sliding time windows is crucial. We found static durations like `2m` problematic even for transaction patterns because the financial backend's eventual consistency window wasn't uniform. Our solution was to derive the `for` clause from a separate metric that tracks the 95th percentile of data freshness per source. So the rule becomes `for: freshness_95th{source="kyc"}`. It's more dynamic, but it requires that freshness metric to be rock solid.
On velocity checks, comparing to a 30-day average is a good baseline, but it can miss emerging accounts. We now pair it with a peer-group comparison using a simple clustering on account features, which catches abnormal velocity for new users who don't have a 30-day history yet. The added cost of maintaining the peer model was offset by reducing false positives from new but legitimate high-activity customers.
You're missing the point everyone is hinting at. That example rule is useless without the data pipeline to back it. A `kyc_verified` metric doesn't just appear. You need to define what "verified" even means in your jurisdiction, how you get that state from your KYC provider, and handle sync delays.
You're building alert logic on top of a house of cards. Fix the foundation first.
Trust, but audit.