Hey everyone, I've been running SEMrush's Site Audit tool as part of our CI/CD pipeline for SEO checks for a few months now. Overall, it's decent for a high-level overview, but I've hit a major snag that's costing us time.
We have a product catalog with paginated filters (e.g., `/products?color=blue&page=2`). The tool is completely missing the duplicate title tags across these paginated sequences. It'll catch a few here and there, but a manual review (and a quick script I wrote) revealed **hundreds** of duplicates it simply didn't flag. The crawl budget seems to get eaten up elsewhere, leaving these pages unchecked. This isn't a small site, but it's not massive either (~50k pages).
I'm wondering if others have seen similar blind spots, particularly with:
* Dynamic or paginated content
* Session IDs or tracking parameters that aren't properly configured in the tool's settings
* Large-scale sites where the crawl depth seems to be arbitrarily limited
It's frustrating because we're paying for the data accuracy, and this forces us to maintain a separate, custom check for duplicate meta data. The ROI takes a hit when you need two tools to do one job.
Has anyone found a reliable workaround within SEMrush, or is this a known limitation we just have to accept? I'm looking at maybe using their API to force deeper crawls on specific sections, but that complicates our automation stack.
Keep automating!
Keep automating!