Alright, let's set the stage. We run a decent-sized e-commerce operation. Not a startup, not a Fortune 500. We've been using a popular, cloud-based MMM tool for about 18 months. The model is supposedly "AI-powered," trained on our ad platform spend, some basic weather data, and our own sales data.
Every single quarter, the #1 recommendation from the model's "optimization simulator" is the same: double down on brand search and video. We're talking a 90-110% increase in that budget bucket. The output looks convincing—fancy charts, a 0.85 R-squared, the whole nine yards.
Meanwhile, Sales and the performance marketing team are screaming. Their lead pipeline is drying up, and they can point directly to weeks where we tested a 20% brand spend bump with zero measurable impact on conversions. Our own first-party data shows the customer journey starting with a competitor brand term or a direct visit more often than our own brand terms.
So we've got a black-box MMM telling us to pour money into a brand halo, and ground-truth sales data saying it's a waste. The tool's vendor just shrugs and says "the model is probabilistic, not deterministic" and suggests we're not being patient enough for long-term brand equity.
Has anyone else had their MMM go full brand evangelist? I'm starting to suspect the issue is in the model's priors or the data inputs. If you feed it mostly marketing mix data, maybe it just learns to value the spend with the highest historical correlation, not actual causality.
```python
# Pseudo-code of what I think is happening
model_priors = {
"brand_spend": "high_equity_long_term",
"performance_spend": "noisy_short_term"
}
# Garbage in, gospel out.
```
What are the concrete levers here? Do we need to feed it more granular conversion data? Fight with the model's assumptions? Or is this just a sign we need to ditch the SaaS black box and run something like Robyn on our own infra, where we can see and tweak the damn knobs?
Just my 2 cents
Been in this exact trench. That 0.85 R-squared is the vendor's favorite sleight of hand, making you think you're looking at truth when you're probably just seeing a model that's overfitting to your overall revenue trend. It's a correlation masquerading as causation.
The real alarm bell is the vendor's "probabilistic, not deterministic" line. That's their get-out-of-jail-free card when their recommendations fail to materialize in your actual pipeline. If your own incrementality tests show a 20% bump did nothing, and your first-party journey data shows prospects starting elsewhere, then the MMM is likely misattributing baseline sales to your brand efforts. It's crediting your brand ads for sales that would have happened anyway.
Have you looked at the model's assumed adstock decay rates and lag effects? I've seen these tools set absurdly long half-lives for digital brand campaigns, like six weeks, which artificially inflates their perceived value. Ask for the raw coefficient tables and the confidence intervals on those brand terms. I'll bet they're wider than they'd like to admit.
MQLs are a vanity metric.