Most enterprise SEO platforms charge by the number of URLs you can crawl per project. If you have a large site, you quickly hit a paywall or an artificial throttle. The official line is "scale," but the reality is vendor lock-in.
You can bypass this by using Screaming Frog's built-in scripting engine. It lets you offload the raw data processing to a local Python script, turning SF into a lean, mean, crawling machine that only holds the logic you need. The expensive platforms are just doing the same thing with a prettier UI.
Here's the workflow:
1. **Crawl with Screaming Frog** in `List` mode, feeding it a sitemap or URL list. Configure it to extract only the specific data you need (e.g., meta tags, status codes, response times) and set the custom extraction to `XPath` or `CSS Path`.
2. **Export the raw data** at intervals or at the end of the crawl via the `Script` tab. You'll typically export the `Internal:All` dataset.
3. **Process with Python** using `pandas`. This is where you apply your business logic, filters, and generate reports, without being limited by the tool's GUI or row limits.
Example script stub to handle a large export and flag slow pages:
```python
import pandas as pd
# Load the raw crawl data exported from Screaming Frog
df = pd.read_csv('screaming_frog_export.csv')
# Filter for slow-loading pages (e.g., TTFB > 1.5s)
slow_pages = df[df['Time To First Byte (ms)'] > 1500][['Address', 'Time To First Byte (ms)']]
# Apply custom logic: group by directory, find missing h1 tags, etc.
# This is where your actual analysis happens, on the full dataset.
missing_h1 = df[df['H1'].isna()]['Address']
# Output results for further action
slow_pages.to_csv('slow_pages.csv', index=False)
missing_h1.to_csv('pages_missing_h1.csv', index=False)
```
The key is that Screaming Frog does the HTTP fetching and basic extraction reliably, while Python handles the heavy lifting on the data. This combo costs the price of Screaming Frog's license and a few hours of scripting, versus thousands per year in SaaS fees. The only real limit is your own hardware.
-shift
shift left or go home
Cute hack, but you're swapping a paywall for a timewall. Python can handle the logic, but you're still bottlenecked by Screaming Frog's actual crawl rate, especially on distributed or edge-heavy sites. I've seen this fall apart when you need to coordinate 200K+ URLs with dynamic content, because SF's scheduling gets... whimsical.
The script engine is useful for post processing, but you're still paying for SF's license and hitting its memory ceiling. For truly large scale, you're better off with a headless crawler framework like Scrapy or Colly, containerized and orchestrated. Then you're just paying for compute, not features you've disabled.
You're absolutely right about the timewall on huge crawls. Where I've found this hybrid approach shines is for ongoing, mid-sized audits where you need a quick, visual overview in SF before pushing the enriched data to a BI tool.
It's less about replacing Scrapy for a 200k URL deep dive, and more about making the 30k URL monthly check-in faster than a full enterprise platform cycle. The script engine lets me flag anomalies live, so I'm not waiting for the whole crawl to finish before seeing red flags.
But yeah, for a one-time massive site migration, I'd be right there with you spinning up containers.
Benchmark or bust
That's a helpful distinction. I've been trying to scope this for our monthly vendor audits, and the 30k URL check-in is exactly the spot I'm in.
Do you run into issues with the live flagging when the script is processing? I'm worried about it missing something if the crawl's still pulling data. How do you manage that sync?
Still learning.