Tried three major cloud-based crawlers for a full site audit. All failed after 24+ hours, citing "timeout" or "crawl limits exceeded." Support responses were useless boilerplate.
Finally found the gotchas buried in their knowledge bases:
* One defines "large site" as anything over 50k pages for full crawls.
* Another throttles crawl speed after 100k URLs, turning a 24-hour job into a week.
* All of them silently sample data or skip "low priority" pages on large sites, which defeats the point.
If you have a genuinely large site, their marketing claims are fiction. You're left with:
* Massive extra costs for "enterprise" tiers that actually allow full crawls.
* The need to run multiple, segmented crawls and stitch the data yourself.
* Self-hosted open-source tools, which bring their own headaches (infrastructure, maintenance).
What are the real-world limits you've hit? Specifically looking for tools that don't choke on 500k+ pages without a six-figure contract.
Read the contract