Two passes is trading time for simplicity, which is valid. But this is where everyone's math goes off the rails.
You're spinning up and managing two full crawl systems. The compute and storage cost for the intermediate "clean URL list" at 500k entries isn't trivial if you're doing it right (storing metadata, status, maybe a hash). And you're still paying for the second crawl's execution time, which, even with a seed list, is a hefty resource block.
The real oversight is assuming the "polite delay" is free. Adding a 1-second delay between requests to be nice to your own server means that second crawl now takes 500,000 seconds minimum. That's nearly 6 days of continuous runtime. Your cloud instance cost for a 6-day continuous run, even a small one, will easily eclipse a month of many commercial tools. People forget to price their own time and the infrastructure carrying cost.
pay for what you use, not what you reserve
Yep, the hidden limits are the worst part, aren't they? Been burned by that "silent sampling" myself. It renders the data useless for actual decision-making.
You're right about the self-hosted headaches, but there's a middle ground I've used successfully: treating the crawl as a pipeline. Use something lightweight like Scrapy for the discovery and fetch of all 500k URLs, dumping raw HTML to S3 or cloud storage. Then, you use a separate, scalable process (like Lambda or a batch job) for the actual analysis and auditing against that known set.
It splits the problem. The crawl becomes a known-quantity data collection job, and the heavy lifting (rendering, SEO checks) can be scaled independently without the vendors' black box. You still have to host it, but it's predictable and you own the data.
Dashboards or it didn't happen.