Walkthrough: How to audit a site's pagination without getting blocked by the crawler.

Pagination audit is a classic crawl-blocking trap. Most off-the-shelf SEO crawlers will either get blocked or miss pages because they don't respect the site's pagination pattern correctly. You need to script it yourself.

Here’s a basic Python script using `requests` and `parsel` that mimics a real browser and adds delays. The key is to find the "next" link pattern first, then follow it programmatically.

```python
import requests
import time
from parsel import Selector

headers = {
'User-Agent': 'Mozilla/5.0 (compatible; AuditBot/1.0; + http://yourdomain.com )'
}
session = requests.Session()
session.headers.update(headers)

base_url = "https://example.com/blog"
current_page = base_url
page_count = 0

while current_page and page_count < 100: # safety limit
response = session.get(current_page)
selector = Selector(text=response.text)

# Do your audit logic here (e.g., check for canonical, index tags)
print(f"Auditing: {current_page}")

# Find the 'next' link - adjust selector for the site
next_link = selector.css('a.next::attr(href)').get()
current_page = next_link

time.sleep(1.5) # critical politeness delay
page_count += 1
```

Gotchas:
* Start with a small sample to identify the correct CSS selector for the 'next' link.
* Always implement a hard stop (`page_count` limit) and a delay (`time.sleep`).
* Check `robots.txt` and respect `Crawl-Delay`.
* Log everything to pick up where you left off if blocked.

This approach gives you control. You can extend it to check for common issues like parameter-based pagination, rel="next", or inconsistent patterns across sections.

—cp