Hey folks 👋
Been spending a lot of time analyzing customer behavior logs from our web app, trying to feed that data into Claude.ai to generate product insights. The goal is to move beyond generic "users like feature X" and get to *actionable* feedback for our dev and product teams.
I've had mixed results. Sometimes Claude gives me a brilliant breakdown of a potential friction point; other times it's just a reworded summary of the data.
From a martech angle, I'm used to structuring data for platforms like HubSpot or Marketo to trigger specific workflows. I'm wondering if there's a similar "prompt architecture" that works well for product feedback analysis.
What's working for you? I'm especially curious about:
* **Log formatting:** Are you feeding raw logs, or doing some pre-processing (like session stitching) first?
* **Prompt framing:** Do you explicitly ask for "actionable insights" or define what that means (e.g., "suggest a UI change," "identify a bug hypothesis")?
* **Examples in prompt:** Have you had success by providing one or two examples of the output format you want?
For instance, I tried pre-grouping logs by user session and adding a line like "Based on the sequence of actions above, what is one hypothesis for a product improvement?" That helped a bit.
Would love to compare notes. How are you structuring this to get the most operational value?
Pick the right stack.
MartechMatch
I'm a backend engineer at a mid-size SaaS shop (120 people) where we run a Python/Go monolith and have spent the last year building a product insights pipeline that actually drives our quarterly planning. We feed about 2TB of application event logs per month into a mix of ClickHouse and BigQuery, then use structured prompts with Claude Opus to generate weekly digests.
Here's a breakdown of what we tested and what finally stuck:
* **Log preprocessing is non-negotiable:** Feeding raw logs gave us unusable noise. We now pre-aggregate logs into user sessions with a 30-minute inactivity threshold, and we attach key metadata like plan tier, feature flags, and A/B test cohort. This stitching step (a Python script that runs before analysis) cut our "vague summary" outputs by about 70%.
* **Prompt framing needs a strict output schema:** Instead of just asking for "actionable insights," we define the exact format. Our prompt includes: "Return a JSON array with objects that have keys: `insight_type` (must be 'ui_change', 'bug_hypothesis', or 'growth_opportunity'), `confidence` (0-1), `evidence_log_ids` (array), and `recommended_action` (one sentence)." This forces the model to operate like a classifier.
* **Provide two concrete examples in-context:** We include two perfect example outputs in the prompt itself, drawn from a past successful analysis. One shows a bug hypothesis from slow API calls followed by a rage-click pattern, and another shows a UI change suggestion from a sequence of hover events and abandoned modals. This few-shot approach improved relevance dramatically.
* **You must limit scope per query:** Asking for "insights from all logs" is a recipe for generic fluff. We now segment by a specific feature or user cohort (e.g., "enterprise users on the new dashboard") and analyze no more than 500 sessions per prompt. This keeps the model's attention narrow and the findings specific. Running 10 focused queries beats one giant one.
My pick is the **structured JSON-output prompt with sessionized logs and two in-context examples**. It's working for our use case of turning event streams into engineering and product tickets. If you're dealing with mostly unstructured support chat logs instead of application events, or if your team needs real-time detection, tell us that - the tooling and prompt approach would shift completely.
System calls per second matter.
You're on the right track with session stitching. That's the bare minimum for signal. The metadata user25 mentioned is critical - without plan tier and feature flags attached, you're analyzing noise.
Your prompt needs to force a decision. I frame mine as: "Act as the product lead. Review these session sequences. For each anomalous pattern, propose ONE of: a bug ticket (with steps to reproduce), a UI/UX change (with mockup description), or a new feature hypothesis (with target segment). Do not summarize."
Provide one clear example of each output type in the prompt. If you don't define "actionable," the model will default to safe, useless summarization.