Everyone talks about crawl budget like it's the holy grail. But I've yet to see a straight answer on actual, measurable speed limits imposed by the major SEO audit platforms. Vendor claims are worthless.
When I configure a crawl in Ahrefs, Semrush, or even Screaming Frog, what's the real throttle? Is it requests per second? Concurrent connections? Do they even publish these numbers, or is it a black box you only hit when you try to crawl a massive site and it takes three days? I care about actual throughput for large-scale enterprise sites. The "unlimited" plans always have a soft cap somewhere.
Just my two cents.
Yeah, the "black box" feeling is real. I've seen the same thing trying to benchmark crawl performance for internal monitoring. Even with tools like Screaming Frog you're often limited by your own hardware/bandwidth, not the tool.
What's wild is the lack of transparency on the SaaS platforms. They'll talk about "unlimited URLs" but never about requests per minute. The cap usually shows up as a vague slowdown after a certain point.
You could try running your own parallel crawls with a custom script and measure the response times/delays from the target server. That might give you a better baseline to compare against the "black box" throttling. It's extra work, but then at least you know.
#k8s
True, with Screaming Frog you're often bottlenecked by your own machine. I learned that the hard way when my old laptop couldn't handle a big crawl.
> try running your own parallel crawls with a custom script
Is there a common beginner-friendly tool for this, like a simple Python library you'd recommend for testing? I've heard of Scrapy but it seems complex for just a quick speed check.
They never publish them. That's the point.
You'll only discover the throttle when your bill hits enterprise pricing. Then suddenly your "unlimited" crawl has a polite, unlisted limit that support will vaguely reference as "system optimization."
The real throttle is their own infrastructure cost. They won't tell you the RPS cap because it's a moving target based on load and your subscription tier.
Just my two cents.