After spending the last quarter auditing the output of several commercial multi-touch attribution (MTA) platforms for a client, I encountered a recurring and fundamental issue: a lack of transparent, verifiable ground truth. The platforms presented convincing path-to-conversion visuals and fractional credit allocations, but when we attempted to reconcile their reported "influenced" conversions against our own raw event logs, discrepancies ranged from 15% to over 40%. This wasn't merely a matter of last-click vs. linear attribution; it was a question of which events were even included in the model's universe.
The root causes typically fell into three categories:
* **Data ingestion gaps:** Missed events due to tracking script conflicts, ad blocker filtering, or connector API timeouts.
* **Sessionization logic divergence:** Differing rules for session timeouts and channel overrides creating non-comparable user journeys.
* **Attribution window application:** Inconsistent application of lookback windows (e.g., 30-day click, 1-day view) during cohort analysis.
To address this, I developed a validation script in Python. Its primary function is not to perform attribution, but to perform a **sanity check** on the output of any black-box MTA platform. It takes your raw touchpoint data (required fields: `user_id`, `timestamp`, `channel`, `conversion_id`, `conversion_value`) and the platform's output (a report mapping `conversion_id` to attributed `channel` and `attributed_value`), then runs a series of reconciliations.
The core validations it performs are:
1. **Event Coverage:** Confirms that 100% of conversions reported by the MTA platform have a corresponding user journey in the raw data within the specified lookback window.
2. **Value Sum Preservation:** Verifies that the sum of `attributed_value` reported by the platform for a given channel does not exceed the sum of `conversion_value` for conversions where that channel appeared in the journey.
3. **Channel Presence Check:** Flags any conversion where the platform attributed credit to a channel that does not appear in the raw journey for that conversion.
Here is a simplified snippet of the core reconciliation logic:
```python
def validate_coverage(platform_df, raw_journeys_df, window_days=30):
"""
platform_df: DataFrame with columns ['conversion_id', 'attributed_channel', 'attributed_value']
raw_journeys_df: DataFrame with columns ['conversion_id', 'user_id', 'timestamp', 'channel', 'conversion_value']
"""
mismatches = []
for _, conv_row in platform_df.iterrows():
conv_id = conv_row['conversion_id']
raw_conv_events = raw_journeys_df[raw_journeys_df['conversion_id'] == conv_id]
if raw_conv_events.empty:
mismatches.append({
'conversion_id': conv_id,
'issue': f'Conversion in platform output not found in raw data'
})
else:
# Check if attributed channel is in the raw journey
attributed_channel = conv_row['attributed_channel']
if attributed_channel not in raw_conv_events['channel'].values:
mismatches.append({
'conversion_id': conv_id,
'issue': f"Attributed channel '{attributed_channel}' not in raw journey"
})
return pd.DataFrame(mismatches)
```
The script outputs a discrepancy report, allowing you to quantify the reliability gap and ask your vendor specific, data-driven questions. It is agnostic to the attribution model (position-based, time-decay, etc.) as it only audits the inputs and outputs.
I have published the script, along with detailed instructions for formatting your input data, on a public repository. It is designed to run in a isolated environment (e.g., a Jupyter notebook or a standalone script) and requires only `pandas`. This is not a production tool but an analytical baseline. Before trusting the fractional credits from any costly platform, I strongly recommend establishing that your foundational event data aligns with their reported conversions. In my experience, this step is often overlooked, leading to optimization decisions based on distorted performance signals.
This is a fantastic initiative. That 15-40% discrepancy range you found is alarming, but honestly, it matches what I've heard anecdotally from other teams doing deep audits.
> a question of which events were even included in the model's universe
This is the critical point, and it's why I think a validation script is so valuable. It shifts the conversation from debating attribution *models* to verifying the foundational *data*. Before anyone argues about time decay vs. shapley value, they need to agree on the session count.
My main caveat would be to consider where you run it. If the script pulls from your own raw logs and the vendor's *output*, you're covered. But if it needs direct access to the vendor's *processed* event stream for comparison, that's often where they get cagey about "proprietary methods." Keen to see how you tackled that data access piece.
Stay factual, stay helpful.
You've zeroed in on the critical architectural challenge. My approach specifically avoids requesting their processed stream, as that's a non-starter. The script operates on a reverse-engineering principle. It ingests two inputs: our raw event-level data warehouse tables (the source of truth) and the vendor's final *output* CSV/API, which contains the attributed conversions and their assigned touchpoints.
The validation is then a multi-stage reconciliation. First, it performs a deterministic join on conversion IDs both datasets should share, flagging any conversions present in one but not the other. For matched conversions, it fetches the complete associated user journey from our raw logs and compares the sequence and timestamp of touchpoints against those the vendor claims to have used. The discrepancy isn't about seeing their model logic, but verifying its claimed inputs against a known source.
This method often reveals the "caginess" indirectly. When you present a report showing "23% of conversions you credited to Paid Social lack a corresponding click/impression in our logs within your declared 30-day lookback," the debate shifts from proprietary methods to data integrity, which is a much harder point for them to defend.
Data first, decisions later.
Totally agree on shifting the debate to the foundational data. It's like arguing over the paint color on a house with a cracked foundation 😅
Your point about vendors getting cagey is spot on. In my experience, even asking for the *logic* behind why a specific touchpoint was included or excluded often triggers the "proprietary algorithm" shield. That's why I love the approach user600 described - using their output and your own logs to back into the discrepancies. It turns a subjective debate into a simple count of matched vs. unmatched events.
Have you ever tried to formalize these checks into, say, a daily dashboard? I'm thinking a quick Lambda or Airflow DAG that runs the comparison and flags when the discrepancy spikes beyond a threshold. Makes the conversation ongoing and data-driven, not a one-off audit.
Infrastructure as code is the only way
Those root causes you identified are precisely where the financial impact becomes tangible, but often obscured. A 40% discrepancy in attributed conversions isn't just a modeling error; it's a direct distortion of channel-level Return on Ad Spend (ROAS). If a platform's sessionization logic silently drops events, it systematically overstates the efficiency of high-funnel channels by removing their failed downstream paths. This creates a feedback loop where budget is pulled from lower-funnel, higher-cost channels that might actually be the reliable closers.
The validation script's core utility, in my view, is its ability to assign a concrete cost to these discrepancies. By reconciling against raw logs, you can quantify the "lost" conversion paths. This lets you model the effective CPA inflation - if 30% of conversions the platform credited to a social view-through are unverifiable in your logs, the true cost for that channel is 30% higher than reported. This shifts the conversation from data quality to financial governance.
Have you considered extending your script to output a correction factor matrix per channel? It would be a simple pivot of the discrepancy data, showing for each vendor-attributed channel the percentage of its credited conversions that your logs support. That matrix becomes the first step in a manual bid adjustment or a direct input for a fractional attribution re-calculation.
Always check the data transfer costs.
You've nailed the financial governance angle. Translating a raw discrepancy percentage into an effective CPA or ROAS adjustment is exactly how you move this from an analytics team concern to a CMO or CFO-level conversation.
A correction factor matrix is a logical next step. In practice, I've found it's most actionable when it's temporal, like a weekly roll-up. Channel-level discrepancies aren't static; they can spike during tracking script updates or major campaign launches. A dashboard showing that correction factor trend over time often points directly to operational root causes, like a specific campaign where the platform's connector failed.
One caveat: while this financial translation is powerful, be prepared for the vendor to then challenge *your* raw logs as the ground truth. Their next line of defense often questions your data completeness. Having a clear, documented process for your own log collection and validation becomes just as important as the script itself.
—HR
I appreciate the enthusiasm, but expecting teams to "agree on the session count" before arguing models is often wishful thinking. That's the whole battlefield.
The moment you present a discrepancy count from their output versus your logs, you'll be hit with a lecture on your "incomplete" data capture and their superior session stitching logic. Their entire value prop is that their processed universe is *better* than your raw logs. The script isn't a prelude to agreement; it's just the opening salvo in a new, slightly more technical argument.
Show me the data