Hey everyone,
I'm in the middle of a multi-phase migration for a client, moving from a monolithic setup on EC2 to a containerized architecture on EKS with CloudFront in front. Since we switched the staging environment over to the new domain and infrastructure, Semrush's Position Tracking has been giving us wildly inconsistent data for the same keywords.
It's like the tool can't decide which domain to track, even though we've updated the campaign settings. We're seeing:
* **"Keyword not found"** for terms we know are ranking, but only on some updates.
* Massive, day-to-day position swings (e.g., from #12 to #45 and back) that don't align with Google Search Console data.
* The "Visibility" score seems completely unmoored from reality now.
Has anyone else hit this after a site migration, especially one involving a domain or subdomain change? I'm trying to rule out if it's:
1. A known Semrush data lag/crawl issue during DNS propagation (even though propagation is long done).
2. Something about our new architecture (e.g., CloudFront headers, different user-agent handling) confusing their crawler.
3. Just a quirk of their tracker that requires a full campaign reset.
I'm leaning towards creating a brand new tracking campaign, but I really want to preserve historical data. Any war stories or proven fixes would be super helpful.
For context, here's a simplified view of our new setup the tracker might be hitting:
```hcl
# Simplified Terraform snippet for our CloudFront distro
resource "aws_cloudfront_distribution" "site" {
origin {
domain_name = aws_lb.alb.dns_name
origin_id = "eks-alb-origin"
custom_origin_config {
http_port = 80
https_port = 443
origin_protocol_policy = "https-only"
origin_ssl_protocols = ["TLSv1.2"]
}
}
# ... rest of config
}
```
-- Amy
Cloud cost nerd. No, I don't use Reserved Instances.
Yeah, it's probably all three of those things at once. Their crawler logic is notoriously brittle with infrastructure changes. I've seen similar nonsense when we just switched CDNs.
The real tell is that it doesn't match GSC. When a tool you pay for contradicts a free Google tool, the paid one is usually the one hallucinating. Their "visibility" metric is a proprietary black box anyway, mostly for sales decks.
A campaign reset might force a fresh crawl, but you'll lose historical continuity. Classic vendor lock-in problem. You either live with the garbage data for a few weeks or torch your timeline. 🙃 Ever tried ahrefs for a side-by-side comparison? Their tracker flaked out less for us during a domain move.
—aB
Ahrefs flaked out less, but their pricing model is its own special kind of penalty box if you ever need to reduce your subscription later. You're just trading one lock-in for another.
The core issue is treating these trackers as a source of truth. They're not. They're proprietary sampling tools with aggressive margin for error, repackaged as "data." When they glitch, it exposes the marketing.
Your point about resetting the campaign and torching the timeline is the real hidden cost. The vendor gets to say the tool "works," but the historical data you paid to collect becomes useless. Convenient for them, expensive for you.
Trust but verify.
Forget the crawl lag. The problem is likely your new architecture.
You mention CloudFront. Did you audit its cache policies and origin request settings for the Semrush crawler's user-agent? If you're blocking or heavily caching their bot traffic, their sampler gets incomplete pages and returns garbage data.
Check your WAF or security group rules too. A too-restrictive rule set from the migration could be silently dropping their probe requests, causing the "keyword not found" errors. GSC works because it's Google's own bot. Third-party crawlers get the locked door.
A campaign reset just papers over the instrumentation failure.
Least privilege is not a suggestion.