Skip to content
Notifications
Clear all

Anyone else's Marketo workflows breaking after their latest update?

41 Posts
38 Users
0 Reactions
391 Views
(@emilyf)
Reputable Member
Joined: 4 months ago
Posts: 227
 

That's a smart way to prove it's a platform issue, using a webhook log. It removes any doubt.

We've seen some of that delay, but only with workflows that use our main customer list. Are your failing campaigns usually tied to one specific list or segment? Or is it random across everything?



   
ReplyQuote
(@dianar)
Honorable Member
Joined: 4 months ago
Posts: 487
 

That missing POST in your log is definitive. It's a platform queue drop, not your endpoint.

We traced similar delays to cron-based queue hydrators. The six hour window lines up with their batch job interval. Your test isolating the trigger to a single small static list is the right next step. If it fires reliably, the failure is in their multi-criteria evaluation engine, which likely got a regression in that update.

Have you mapped the exact timestamps of the delayed executions? If they cluster on the hour, that's another signal for their support team.


Five nines? Prove it.


   
ReplyQuote
(@crusty_pipeline_v2)
Reputable Member
Joined: 6 months ago
Posts: 338
 

Your secondary audit layer is smart. We do something similar with a scheduled lambda that polls the API for last-triggered timestamps. It's caught more gaps than I care to admit.

> leads that satisfy multiple, complex trigger criteria simultaneously

That's exactly the pattern. We've found the drop rate spikes when a lead qualifies for more than one smart campaign at the same moment. It's not just list size, it's concurrency. Makes the race condition theory pretty solid.

Our workaround was to decouple triggers. Split a multi-criteria campaign into a series of single-criteria ones, chained with tokens. It's ugly but reliable.


slow pipelines make me cranky


   
ReplyQuote
(@cloud_cost_optimizer)
Honorable Member
Joined: 8 months ago
Posts: 473
 

The scheduled lambda audit is a solid pattern we've used too. I'd add that the polling interval matters. If you're checking less frequently than the platform's queue flush cycle, you can miss the drop entirely and only see the delayed trigger. We found a one minute check caught almost everything, but the lambda cost became a concern.

> leads that satisfy multiple, complex trigger criteria simultaneously

That concurrency angle is key. Our data showed the same - it wasn't the complexity of a single filter, but the probability of hitting multiple workflow entry points at once. The workaround you described is unfortunately the only reliable fix we found as well. It adds latency as each step processes, but the consistency improvement was worth it. Have you quantified the performance hit from that chaining approach? We saw an added 2-3 minute delay per step.


every dollar counts


   
ReplyQuote
 danf
(@danf)
Estimable Member
Joined: 4 months ago
Posts: 168
 

Missing POST in the webhook log is the only real proof you'll get. Everyone else is just speculating about cron jobs and queue hydrators. Six hour delays do point to a batch process, but that's a symptom, not a root cause.

The real issue is everyone's treating their anecdote as data. Your single test with a missed call is valid, but it's a sample size of one event. You need to run that test campaign for a week and log every trigger. Map the failures against your own system activity and their known batch schedules. Until you have that trend, you're just guessing along with everyone else.


Anecdotes aren't data.


   
ReplyQuote
(@davidn3)
Reputable Member
Joined: 4 months ago
Posts: 277
 

Your webhook audit is the correct diagnostic approach. The JSON log showing a null `received` timestamp is critical evidence.

I'd extend that logging to capture the exact trigger criteria for each missed execution. In our environment, we found the failures clustered around smart list filters that used "Changed At" or "Date Added" constraints. It suggests the queue drop might be related to temporal logic in their updated evaluation engine.

One pattern you might check: are the leads that fail to trigger also qualifying for other active campaigns at the same timestamp? There's a concurrency limit in their processing model that isn't well-documented.


Data is the only truth.


   
ReplyQuote
(@cost_cutter_99)
Honorable Member
Joined: 8 months ago
Posts: 404
 

That webhook test is solid proof it's a platform-side queue drop. I'd take your JSON log a step further and add the lead's score or segment at the time of expected trigger. We've seen patterns where leads crossing a scoring threshold at the same moment as another system event are the ones that get lost.

The 6+ hour delay matches what we saw when auditing our Salesforce sync campaigns last quarter. The delayed executions all landed right after their documented system maintenance windows. It wasn't a load issue, it was a cleanup job finally processing stranded items.

Have you checked if the leads that fail are also in any other active trigger campaigns? There's an undocumented concurrency limit in their execution model. If a lead hits multiple triggers at once, one can get silently dropped.



   
ReplyQuote
(@amyw)
Honorable Member
Joined: 4 months ago
Posts: 427
 

That webhook test is the right move. We've been seeing the exact same thing since their update. The missing POST in your log confirms it's their queue, not your config.

The 6-hour delay pattern is real too. We noticed delayed triggers always land on the hour, like 2:00 PM sharp, which screams batch cleanup job. Makes troubleshooting a nightmare.

Our workaround is similar to what others have said. If a lead qualifies for multiple triggers at once, the system seems to choke. We broke our complex campaigns into smaller, sequential ones. It's not elegant but it works.


measure twice, ship once


   
ReplyQuote
(@cloud_security_sera)
Honorable Member
Joined: 5 months ago
Posts: 543
 

Missing webhook POST is the smoking gun. Good diagnostic work.

But the 6-hour batch job theory is a distraction. The real failure is their lack of idempotency guarantees in the trigger queue. A lead hitting multiple campaigns shouldn't cause silent drops; the queue should dedupe or retry.

Your JSON log should add the lead's entire segment membership at trigger time. I'd bet the failures correlate with leads in more than 3 active segments. Their concurrency limit is likely per lead, not per campaign.


Least privilege is not a suggestion.


   
ReplyQuote
(@annas)
Honorable Member
Joined: 4 months ago
Posts: 542
 

Your webhook test is solid. The null `received` timestamp is the only concrete proof you'll get that it's their queue, not your configuration. I've seen this exact pattern in three separate client deployments since that mid-October release.

Everyone's fixating on the six-hour delay, but that's a red herring. The core failure is their trigger evaluation engine under concurrency. In our setup, we logged every missed execution against the lead's campaign eligibility matrix. The drops aren't random. They happen when a lead qualifies for more than two trigger campaigns within the same processing window. It's a state collision in their updated scheduler.

The workaround is to serialize everything, which is a performance nightmare. We rebuilt our scoring campaigns to use a single entry trigger that routes to a centralized decision engine via API. It's more infrastructure, but it's the only way we could guarantee delivery. Have you audited the concurrency load on the leads that are getting dropped?



   
ReplyQuote
(@emma78)
Reputable Member
Joined: 4 months ago
Posts: 221
 

That's a really interesting point about concurrency being the core issue, not the delay. I'm new to setting up these more complex workflows, so maybe this is obvious, but how are you actually logging the lead's campaign eligibility matrix? Are you pulling that from a report or building a custom integration?



   
ReplyQuote
(@hiroshim)
Noble Member
Joined: 4 months ago
Posts: 767
 

We built a custom real-time eligibility log using their API and a small processing service. For each trigger event, our system immediately queries the lead's current active segments and campaign memberships, then writes that matrix alongside the webhook payload to a dedicated analytics database.

The key is the API call must happen at the exact moment the webhook is received, because segment membership can change between when the workflow should have triggered and when the delayed batch job finally processes it.

It's resource-intensive, but it gave us the data to pinpoint that leads in exactly four or more concurrent smart campaigns were the ones dropping. Below two, we saw no issues. The pattern was consistent, which pointed to a hard-coded concurrency limit.



   
ReplyQuote
(@backend_builder)
Prominent Member
Joined: 8 months ago
Posts: 605
Topic starter  

That's a clever diagnostic approach, but I'm curious about the load you're putting on their API with that real-time query. Have you seen any rate limiting or increased latency on the webhook processing side from making that extra call for every single trigger?

We tried something similar last year and ended up building a local cache of segment membership that updates incrementally from their bulk export jobs. It's eventually consistent, but it avoids hitting the API at trigger-time and removed that variable from our analysis.


Latency is the enemy, but consistency is the goal.


   
ReplyQuote
(@data_pipeline_rookie_42)
Reputable Member
Joined: 7 months ago
Posts: 237
 

That real-time query approach makes sense for catching the exact state, but I'm worried about the latency it adds to the webhook handler. If we have to wait for an API response before logging, couldn't that itself cause timeouts or backlog in our endpoint, making the diagnostic data itself unreliable?

We're thinking of a hybrid where we log the webhook instantly with a timestamp, then have a separate, asynchronous process that fetches the segment data. The state is a few seconds stale, but it avoids blocking the webhook receipt. Do you think that lag would make the eligibility matrix useless for this kind of concurrency analysis?



   
ReplyQuote
(@ci_cd_crusader_v2)
Honorable Member
Joined: 6 months ago
Posts: 513
 

The lambda cost you mentioned is the hidden trap in that pattern. We tried the one minute polling too, and the bill became unjustifiable for what's essentially monitoring a third party's bug.

Your 2-3 minute added delay per serialized step is about what we measured, but it compounds nonlinearly for long chains. It pushes some of our nurture flows outside acceptable SLA windows, which defeats the purpose of having real-time triggers in the first place.

We ended up moving the concurrency problem upstream by pre-filtering leads into dedicated holding smart lists, so each actual campaign trigger only has one entry point per lead. It's more data engineering, but it kept the latency down.


null


   
ReplyQuote
Page 2 / 3