Oh that's the good stuff right there. I've been living in the Ahrefs API docs for a project with a similar structure. Your point about automating profile creation is key - nobody's got time for manual entry when you're scaling.
One caveat I'd add to your three key considerations: you have to think about *profile naming conventions* in your automation script from day one. If you're spinning up profiles for `clientX.platform.com` and `clientX.platform.com/uk/blog`, you need a predictable naming logic that both the API and your team can parse. I once ended up with "ClientA_Profile_1," "ClientA_Blog_UK," and "ClientA-UK-Blog" for the same entity because the scripts evolved separately. Total mess.
How are you handling the actual keyword assignment? Are you running each keyword list against all profiles and letting the attribution rules sort it, or pre-sorting keywords into buckets before they even hit the tracker? I've tried both, and the pre-sort saves API calls but the post-sort is more accurate.
If it's not measurable, it's not marketing.
Naming conventions are a maintenance nightmare if you don't lock them down early. We use a strict template from the API script: `{client_id}_{site_segment}_{purpose}`. The `site_segment` is parsed from the URL pattern, never manually entered.
> pre-sort saves API calls but the post-sort is more accurate
That's the trade-off. I pre-sort with regex patterns to cut costs, but run a monthly post-sort audit on a sample. The pre-sort misses about 5-10% of edge cases, but it's good enough for weekly reporting. The full post-sort is for monthly deep dives.
Good on you for digging into the automation from the start. That Python snippet you're hinting at is going to save you weeks of tedious clicks.
Your third point on keyword mapping is the real hairball. When a keyword ranks for multiple URL patterns, most tools will just dump it into the first matching profile's bucket, which paints a completely wrong picture of what's actually driving conversions. You have to enforce a hierarchy in your attribution logic, like subdomain > key subdirectory > root, and even then, you'll need a quarantine bucket for terms where the ranking URL flips weekly.
And a warning on those APIs: their rate limits are often laughably low for a multi-tenant setup. Don't be surprised if you have to architect your script around aggressive batching and caching, or you'll be waiting all day for data.
keep it simple
Love that naming template. Enforcing it via the API script from the start is the only way it sticks. We learned the hard way that letting even one person manually name a profile creates a fractal of inconsistencies.
That pre-sort/post-sort balance is smart. We do something similar, but we also flag any keyword that gets caught in the post-sort correction for manual review. Those edge cases often uncover a new URL pattern we should be tracking or a SERP quirk worth noting.
Do you ever find that the missed edge cases cluster around a particular type of subdirectory, like date-based archive pages or tag pages?
Stay constructive
Hierarchy is solid, but you're still left with a cost problem. That `keyword_url contains` check runs for every keyword, every update. For a multi-tenant setup with tens of thousands of keywords, those string operations add up in compute time and API cycles.
Better to pre-compute a lookup table of URL patterns to profile IDs on a schedule, then just match against that. It's cheaper.
cost per transaction is the only metric
That's a solid starting list of considerations. But the moment you said "automate profile creation," my mind went straight to the API's hidden landmines, especially for the multi-tenant piece.
You'll be fine spinning up the profiles. The real test is when your first client churns, or you sunset a regional subdirectory. If your cleanup script isn't as robust as your creation logic, you're left with phantom profiles piling up and eating budget.
Did you build a decommissioning flow into your automation from day one, or is that a "future us" problem?
Spreadsheets > marketing slides.
Great question. The cleanup script was actually the first piece I wrote, because those phantom profiles cost real money 😅 I found that most APIs don't give you a clean "delete_by_client" method, so you need to map active subdomains/directories from your own DB and then reconcile.
My script runs a weekly diff and moves orphaned profiles to a "graveyard" project. But even then, some budget got spent until I built in a 2-week retention hold for new profiles, in case of client flip-flopping.
Self-host or die trying.
Oh, logging the *ranking URL* change is such a good idea, I wouldn't have thought of that. It makes the reason for a rank drop instantly visible.
That "near SERP feature" flag is a clever next step. I'm wondering, how would you even detect that automatically from an API response? Do you parse the SERP snippet for "People also ask" or check for extra links? I'm new to this level of detail and trying to picture how you'd build that logic 🤔
We profile everything, but we only *report* on high-traffic subdomains after the first data pull. The initial automation batch-creates profiles for all patterns. Then a cron job runs daily, checking the last 30 days of Google Search Console data via its API to filter for subdomains with non-trivial traffic. That list becomes the active set for weekly rank checks. It's a cheap way to avoid API bloat while capturing the long tail - a dormant subdomain can suddenly get traffic from a single viral piece, and you'll catch it on the next cycle.
--perf
That's a clever use of the Search Console API to manage the active set. I've done something similar but with click-through rate as the secondary filter, not just raw traffic. A subdomain might get impressions but if the CTR is abysmal, ranking there might be a red herring for content quality issues.
The daily cron job is aggressive, though. Doesn't that burn through your Search Console API quota pretty fast when you're checking 30-day aggregates for hundreds of potential subdomains? I found I had to run that reconciliation weekly to stay within limits.
That keyword attribution hierarchy sounds clean in theory, but it breaks the moment you try to apply it to a real enterprise contract. Most tracking tools bill per keyword, and their definition of a "keyword" is one unique string in one profile.
If you're duplicating the same keyword across your root, subdomain, and subdirectory profiles to follow that logic, you're triple-paying. The API might let you automate the setup, but the vendor's pricing model absolutely punishes you for trying to get accurate data. I've seen bills balloon from architecture like this.
— skeptical but fair
Nail on the head. The real cost is never the API calls, it's the per-keyword fee they bury in the contract's appendix. They'll happily sell you "granular data," then invoice you for the granularity.
The only way I've made this work is to track the root domain's rankings for most keywords, then use the Search Console API to pivot and attribute clicks/impressions to the actual subdirectory or subdomain. You lose some real-time tracking, but you don't go bankrupt.
You're still paying the vendor, but you're not paying them three times for the same data.
Your stack is too complicated.
Your point about `keyword_url contains` logic is spot on, it's a classic N+1 query problem in a different form. Even with a pre-computed lookup table, you've got to watch out for pattern precedence. What happens when `/uk/blog` and `/blog` both match? You'll need a deterministic rule, like "longest URL pattern wins," to avoid attribution flapping between your profiles.
I've also seen people try to solve this by tagging keywords with metadata at insertion time, like the target profile ID, which moves the cost upfront and makes the rank update a simple join. But that pushes complexity to your ingestion pipeline. It's a trade-off between update-time and insertion-time processing.
Prod is the only environment that matters.
Absolutely correct on the precedence rule. We settled on "most specific pattern wins," which in practice is the longest matching path, but had to formalize it for edge cases like query parameters. We built a simple pattern-matching service that sorts potential matches by a specificity score: path segments count higher than query params.
The metadata tagging approach you mentioned is interesting, but it assumes a static mapping. In our case, the URL structure could evolve, so we still need the update-time reconciliation to handle a keyword's target profile changing, which adds its own layer of complexity to the ingestion pipeline. You're just choosing where to put the mapping logic.
- Mike