Just had to tackle this exact issue last week. Our staging site was getting hammered by scrapers using perfect-looking `User-Agent` strings—think "Mozilla/5.0 (compatible...)"—but their behavior was all wrong. They'd hit the same endpoints in rapid succession, ignoring `robots.txt`. Cloudflare's managed rules didn't flag them because the headers looked legit.
The trick is to combine signals. I built a custom rule in the WAF that catches this pattern. Here's the logic I used:
`(http.request.uri contains "wp-json" or http.request.uri contains "api") and (http.user_agent contains "Mozilla") and (not cf.bot_management.bot_score gt 30) and (http.request.uri in $bad_paths)`
The key parts:
- Target your specific, high-value endpoints (like API routes).
- Check for a browser-like `User-Agent`.
- Use `cf.bot_management.bot_score` to exclude likely-good bots. A score <= 30 is suspicious if they're pretending to be a browser.
- I also use a static list (`$bad_paths`) of already-probed, sensitive URIs to add another layer.
Set the action to *Block* or *Managed Challenge*. It cut the junk traffic by about 90% for us without affecting real users. The beauty is it's cheap, fast to deploy, and you can tweak it based on what you see in your security analytics.
Hope this saves someone else an afternoon of head-scratching! What other signals are you all using to catch sneaky scrapers?
—j
Trust the trial period.
This is clever, but I'm a little skeptical about the bot_score threshold. I've seen legitimate, older browser versions from corporate networks dip below 30, especially if they're behind a shared proxy. Blocking them on your API routes could lock out real, clunky users.
Maybe add a geolocation bypass for known traffic regions? Or pair it with a rate limit on those endpoints instead of an outright block. The $bad_paths list is the real genius part though, that's a solid signal.
Trust but verify.