Hey everyone. Jumping in because I've been testing Lindy for a few weeks now, trying to set up a basic sales outreach and meeting booking agent for our startup.
I *love* the concept, and I've gotten a simple "send follow-up email" workflow to work. But when I try to build something more nuanced—like an agent that can handle a potential customer's specific questions about our pricing tiers based on a PDF I uploaded—the whole "training" phase feels like a black box. I tell it to "learn" from the document and give it some example interactions, but then in practice, the agent sometimes nails it and sometimes goes completely off the rails, quoting things that aren't in the doc.
My main issue is the lack of feedback. There's no way to see *what* it actually learned from the document or which part of my example conversation it's latching onto. It's just "train" and then "test," with a lot of guesswork in between. On a startup budget, I can't afford hours of trial and error with a live customer on the line.
Is anyone else running into this? I'm used to more traditional chatbots where you can at least see a decision tree or confidence scores. With Lindy, I feel like I'm just hoping it generalizes correctly from my few examples. Have any of you found a methodology or a set of best practices that makes this training process more predictable? I really want this to work, but the opacity is making it hard to trust for anything beyond simple tasks.
Would appreciate any insights from the community!
learning daily
You're hitting on the fundamental difference between rule-based systems and these LLM-based agents. With a traditional chatbot, you're right, you can trace the logic. Here, "training" is often just injecting your PDF as context into a prompt behind the scenes.
The inconsistency you're seeing is likely due to that context getting lost, truncated, or the model making things up when it's uncertain. It doesn't "learn" in a permanent way like retraining a model, it just gets better instructions.
For something as critical as pricing, I wouldn't rely solely on an opaque document ingest. You need to bake that data into the agent's core instructions explicitly. Use the document for background, but put the exact pricing tiers, conditions, and limits directly in the system prompt. That gives you a controlled source of truth.
Without visibility into how the agent chunks and recalls your PDF, you're flying blind. It's the observability problem all over again. No metrics, no traces.
You've put your finger on a critical vendor risk management issue, even if you're thinking about it from an operational perspective. The opacity isn't just a UX problem, it's a compliance and audit red flag.
> there's no way to see *what* it actually learned
Precisely. In a SOC 2 or ISO 27001 context, if you're processing customer data or even internal commercial data like pricing through this "training," you need to demonstrate control over that data flow. You currently cannot. You can't audit what the system has ingested, what it retains, or prove it hasn't hallucinated sensitive data into its responses. User202's suggestion of baking critical data into explicit instructions is a compensating control, but it treats the symptom, not the disease.
Your comparison to traditional systems with traceable decision trees is apt. For any regulated data, you must demand explainability. If the vendor can't provide a mechanism to audit the agent's "knowledge" state, you're introducing an unacceptably high risk.
trust but verify