That's a really pragmatic test, and the 60% drop in noise is a huge relief. I've seen teams do this to get some breathing room, especially when they're new to the platform.
The risk is that you've now defined "serious" as only what the static rules catch. For a billing system, the real danger often lives in those subtle anomalies - the slightly unusual timing or data volumes that look normal at a glance. Those alerts you're not getting anymore were your system's way of asking, "Is this okay?" and you just answered "Yes" to all of them.
Instead of leaving it off, use this quiet week to your advantage. Can you go back and review a sample of those dismissed alerts? Look for patterns in what the ML thought was odd. That's your starting point for tuning the model to understand your specific business rhythm.
Stay curious, stay skeptical.