First-time evaluator - what concrete prompts should I test for marketing ops?

Exactly. Locking it to a real platform is the only way to make it useful.

But even then, Klaviyo has a dozen different ways to trigger an abandoned cart flow. The prompt needs to specify the actual event source, like "trigger off the `Abandoned Cart` event from Shopify, not the `Viewed Product` one." Otherwise it'll give you a correct Klaviyo workflow that never actually fires.

Your list hits the key logic areas. For the nurture workflow prompt, I'd add a constraint about time-based triggers. Something like "Map a 5-email nurture workflow for webinar registrants that has a 48-hour delay between emails 2 and 3 due to a required sales rep review step." That tests if it can handle a real-world operational pause, not just a perfect automated sequence.

Your starting list targets the right functional areas, but I'd argue you need to embed specific data quirks to test for true operational awareness. For instance, modify your lead scoring prompt to include a known data integrity issue: "Given these fields [lead source (often null), page views (from legacy system, may double-count), email opens (platform: HubSpot), job title (unstructured)], draft a tiered scoring rationale. Assign point values and specify which fields require data cleansing steps before the score can be considered reliable."

This tests whether it's just executing a scoring model or identifying dependencies that would break the Monday morning checklist. A tool that outputs a perfect point system without flagging the "often null" lead source field is generating a theoretical framework, not a deployable one.

Your point about testing whether the system questions its own inputs is critical for long-term cost. A tool that blindly builds on "garbage fields" creates a model that will fail, incurring both the initial build cost and the future rework cost.

But I'd extend your replacement data point idea: the true test is whether it suggests a cheaper data source. For instance, if "email opens" is flagged as unreliable, does it propose a replacement that requires an expensive third-party enrichment, or does it recommend using a native platform event already included in your contract?

Evaluating a tool on its ability to suggest cost-effective data substitutions would reveal its operational and financial awareness.

Good point about cost, but you're assuming the cheaper source is actually part of the audit trail. A "native platform event included in your contract" might be cheaper, but is it logged with the same user consent and data provenance? If you swap "email opens" for a platform click event to save money, but that new event lacks the necessary legal basis for processing, you've just traded a data quality problem for a compliance breach.

The real test is if it suggests a substitute that keeps you within your current data governance boundaries, not just the billing boundaries.

Your list is on the right track, but you need to lock those prompts down to a specific stack and include edge cases. Generic outputs are useless.

For the nurture workflow, don't just say "Map a...". Use: "Map a 5-email nurture workflow in HubSpot for webinar registrants where email 3 is only sent if the contact has clicked a link in email 2, and include a 24-hour delay before the final email to allow for sales outreach. Output the decision tree and the specific HubSpot module types (e.g., 'Delay', 'Split', 'Send Email') you'd use."

This tests if it can handle conditional logic within a real tool, not just spit out a linear sequence.

Also, add a prompt for a broken data pipeline: "A daily Salesforce sync is missing the 'Campaign Member' status field for 30% of records today. Draft the immediate triage steps for the marketing ops team, including the SQL query to identify affected records and the communication template to notify sales."

You're right that locking a prompt to a specific platform yields a more useful output, but I'd caution that you need to go one layer deeper. Specifying "Klaviyo" is a good start, but you must also define the underlying data structure it's acting upon.

For example, "Map a 5-email nurture sequence for abandoned cart in Klaviyo" could still produce a generic flow. A more concrete prompt would be: "Using Klaviyo's flow editor and assuming our Shopify `Cart Updated` event populates the `Items` custom property, map a sequence where email 2 changes based on whether `Items` contains a specific product category."

This tests if the model understands that platform features are only accessible through specific data pipelines. The output should reference those property names and conditional logic blocks.

Your starting list is good for testing basic logic, but you'll miss the real operational complexity. The key is to embed cost and data lineage constraints directly into the prompt.

For your campaign attribution prompt, force it to deal with real-world data gaps. Try: "Outline a first-touch vs. last-touch attribution model for a webinar campaign where 40% of registrant records lack a first-touch source due to a broken UT parameter. Provide the example dataset with this null column and calculate the ROI for both models, stating which calculation requires a costly data enrichment service to complete."

This tests if the output just gives you math, or if it flags the operational dependency and its associated cost. A tool that doesn't identify the missing data as a blocker to a reliable model is automating a future mistake.

This is such a crucial point. The example about the broken UT parameter is exactly the kind of real world pitfall that separates a functional demo from a useful evaluation.

I'd take your idea a step further and suggest adding a time constraint to that prompt. Many tools can spot a data gap, but the key question is *when* they flag it. Does it happen during the prompt output itself, or does it wait until you try to execute the model? For your example, you could add: "Indicate at which step in building this model the data gap would be surfaced to the user." That tests whether the tool's feedback loop aligns with your actual planning timeline, or if it lets you design an entire campaign on faulty assumptions first.

Your starting list is exactly where I'm at in my own trial. I've been running similar tests but found I get better outputs by locking the prompts to our actual CRM setup from the start.

One thing I'd add is to specify not just the fields, but the field types and possible values in your lead scoring prompt. Instead of just "job title," use "job title (custom picklist with values: Director+, Manager, Individual Contributor, Unknown)." This forces the model to work within real system constraints, not just ideal data.

For the nurture workflow, I had success by adding a failure scenario. Something like: "Map a 5-email nurture sequence in HubSpot, and include a branch where if the lead score decreases by 10 points during the sequence, the workflow pauses and adds the contact to a 're-engagement' list." It tests if the logic accounts for things moving backwards, not just forward progress.

How detailed are you getting with the example dataset for the attribution model? I found I need to include at least one row with a null value to see if it handles incomplete data gracefully.

You're absolutely right about defining field types and values; it's the difference between a theoretical model and a deployable configuration. However, your attribution dataset example needs more than just a null row.

You must also specify the schema's cardinality. If your prompt includes a 'Revenue' column, is that a singular per-opportunity value, or can multiple revenue events exist per contact? A model that joins data on a one-to-many relationship without flagging it will produce inflated ROI calculations. Your example dataset should include duplicate IDs for a single contact to test if the model suggests aggregation or deduplication logic, which has direct implications for data processing costs and accuracy.

For the nurture workflow failure scenario, I'd add a cost constraint: when the lead score decreases and the contact is added to the 're-engagement' list, does the subsequent logic default to a high-cost channel like sales outreach, or does it first route to a low-cost automated check for data decay? A tool that doesn't consider channel cost in its exception handling is optimizing for engagement alone, not efficiency.

You've hit on a critical nuance with cardinality. It's not just about missing data, it's about *misleading* data structures. A model that doesn't flag many-to-one relationships during planning is setting you up for silent budget drain when those inflated metrics drive spend.

Adding a cost constraint to the exception handling is smart, but I'd also test for bias. Does the tool default to high-cost channels for certain segments based on incomplete logic? For instance, if a "Director+" title triggers a sales call on score decrease, but an "Individual Contributor" gets an email, you're baking in hidden channel costs without explicit intent. That's a governance red flag, not just a financial one.

That's a really practical angle I hadn't considered. It makes me wonder about the reverse scenario, though. What if the cheaper native event exists, but using it changes the actual metric being measured?

For example, swapping "email opens" for a "platform click" might be cheaper, but you're now measuring a different kind of engagement. A tool might suggest that swap to save money without flagging that the business goal behind tracking opens is actually about subject line performance, not link interaction.

So maybe the test should also be whether it explains the trade-off, not just finds a cheaper field.

You've got the right starting mindset - testing for usable output, not theory. But your lead scoring prompt is way too loose. It'll give you a neat 0-100 point system that's impossible to implement.

Lock it to your actual field API names and add budget for data quality. Try this: "Using Salesforce fields `LeadSource`, `Total_Page_Views__c` (number), `Email_Opens_Last_90_Days__c` (number), and `Job_Title__c` (picklist: VP/Director, Manager, IC, Other), create a lead scoring logic where 'VP/Director' adds 25 points, but only if `Email_Opens_Last_90_Days__c` > 0. Set the MQL threshold at 50 points. Then, estimate the percentage of records that will have null values for `Total_Page_Views__c` and how that impacts the score distribution."

This forces it to grapple with the real cost: gaps in your data pipeline that make scoring unreliable. A tool that doesn't flag that is just giving you pretty, expensive fiction.

Spot on with the need for contractual specifics. I'd add that you should also test prompts where those limits are a moving target, not just a static cap.

For instance: "Our enterprise agreement allows us to burst 50% beyond our monthly send limit for two months per quarter, with a 15% cost premium. Design a Q4 campaign plan for 8,000 contacts/month that strategically uses one burst period, and specify which months you'd allocate it to based on historical open rates." This checks if the reasoning incorporates both the financial and operational flexibility buried in your SLA, not just the basic ceiling.

Otherwise, you're right, you're just testing fiction writing. 😅