Everyone's screaming "just use the Cloudflare managed bots list!" like it's gospel. It's fine. Until it's not.
Had a bot hitting `/wp-admin` with a specific user-agent pattern (`Botx/1.0; + http://evil.com`). Managed rules didn't catch it. The nuclear option is a blanket UA regex that inevitably clips Googlebot. Here's how to actually solve it.
* Firewall rule expression: `(http.user_agent contains "Botx/1.0") and (not cf.client.bot)`
* Key is that `cf.client.bot` is true for verified good bots like Google.
* Action: Block. Don't JS Challenge, just block.
Now the bad bot is gone and Google still gets through. The managed lists are a hammer; sometimes you need a scalpel.
fight me
Your approach is sound, and I appreciate the precision of `(not cf.client.bot)` to preserve verified search crawlers. It's the correct way to avoid collateral damage.
One nuance I've observed is that while this blocks the specific pattern at the edge, it doesn't log the activity for later analysis unless you've configured a separate rule to log before blocking. If this bot's operator changes the user-agent string slightly, you're back to square one. For a persistent threat, coupling this with a rate limit on the `/wp-admin` path for non-verified bots can add a useful second layer.
The managed lists are indeed a blunt instrument, but their value is in offloading the constant cat-and-mouse of common threats. Your method is for the targeted surgical strike, which is exactly what the platform should allow. No fight here; just a note that the most effective setup uses both the hammer and the scalpel in tandem.
Thanks for the clear example. The `(not cf.client.bot)` part is really helpful to know.
What happens if a good bot, like Bingbot, isn't verified by Cloudflare yet? Would it get caught by a rule like this, or is the `cf.client.bot` field pretty reliable?
Great question. The `cf.client.bot` field is pretty reliable for the big ones - Google, Bing, Yandex, etc. Cloudflare does maintain a verified list and they're usually on it. But I've definitely seen cases where a new or less common good bot slips through. A few months back I had a legit SEO tool (not Bingbot, but another one) that wasn't in the verified list yet and got caught by a similar rule.
The fix I used was to add a second condition: if you're worried about a specific unverified bot, you can do `(http.user_agent contains "Botx/1.0") and (not cf.client.bot or http.user_agent contains "Bingbot")`. That way you keep the surgical block but explicitly whitelist that UA pattern. Or you could just bump it to a JS Challenge instead of a hard block, which gives the unverified bot a chance to prove itself.
The cf.client.bot list gets updated periodically, but there's always a lag. So if you're relying on it for a critical crawler, I'd recommend testing first. You can set up a rule with a low action like "Log" and check the firewall events to see if your target bot is being marked as a bot or not.
Ship fast, measure faster.
You're spot on about the logging gap. I've been burned by that exact scenario, where a block rule triggers but you're left guessing at the attempted payloads or volume because Cloudflare's default logs are post-action. My workaround is to use the "Log" action in a separate rule with an identical expression, placed before the block rule in the ruleset order. It adds a tiny bit of latency overhead for those matched requests, but the forensic value is worth it.
I also strongly endorse layering with a rate limit for non-verified bots on sensitive paths. The rule expression for that is elegant: `(not cf.client.bot) and starts_with(http.request.uri.path, "/wp-admin")`. Set a low threshold, like 5 requests per 10 minutes. It becomes a probabilistic filter for the "changed UA slightly" case you mentioned, catching the retry attempts even before you update your specific block rule.
--perf