I've been using ChatGPT to help train our internal sales engineers on a complex DevOps observability platform, and I've hit a wall. While it's fantastic for generic "overcoming objections" or "discovery call" role-play, it falls apart when we need to simulate a deep-dive technical evaluation with a prospect who has a real-world, messy infrastructure.
The problem is the lack of specific, contextual knowledge. For example, if I prompt: *"Act as a skeptical platform engineer from a large e-commerce company. You are concerned about the overhead of our agent-based distributed tracing."*
ChatGPT will generate reasonable, but *generic*, concerns about performance and resource usage. In reality, the conversation needs to go several layers deeper, immediately. A real platform engineer would ask:
* "How does your agent handle multi-tenant Kubernetes clusters where we deploy pods with `Istio` sidecars already injecting their own proxies?"
* "Can your sampling strategy be dynamically controlled via a ConfigMap we can manage through our existing ArgoCD pipelines?"
* "Show me how the telemetry data flow looks when we have a service mesh *and* your agent, and we need to propagate W3C trace context through our legacy RabbitMQ workloads."
The ChatGPT role-play can't engage on that level without me manually feeding it immense amounts of our own product documentation and common customer architecture blueprints first. It lacks the "ground truth" of our product's specific constraints and the intricate realities of modern hybrid cloud environments.
I've tried to build a custom GPT with uploaded docs, but even then, the role-play feels stilted. It might parrot a feature, but it can't *reason* about the implications in a novel scenario like a seasoned sales engineer can.
My current workaround is to script very specific scenarios, which defeats the purpose of dynamic training. Here's a snippet of the kind of detailed prompt I now have to write to get something marginally useful:
```markdown
**Role:** Prospect, Maya (Lead SRE at FinTech).
**Context:** They have a sprawling GKE setup with Knative for event-driven processing. They use OpenTelemetry Collector but have gaps in Kafka topic instrumentation.
**Technical Stance:** "Your marketing says automatic instrumentation, but we had to fork the OpenTelemetry Java agent for our custom framework. How would your solution avoid that?"
**Known Pain Points:** Can't correlate a user transaction across a Kafka stream, a batch job in Cloud Run, and a legacy Datastore operation.
**Goal of Exercise:** Get the trainee to diagram a data flow and explain how we'd inject context into the Kafka message headers.
```
This is effective but incredibly time-consuming to create for every potential scenario.
Has anyone else tried to use ChatGPT for advanced technical sales or pre-sales training, particularly for complex, infra-heavy products? Have you found a method or a toolchain that makes these simulations more authentic and less generic? I'm wondering if combining it with a RAG setup on our internal architecture diagrams and past deal notes is the only path forward.
— francesc
— francesc
You've nailed the core limitation. The model can't simulate the accumulated tribal knowledge and unique scar tissue of a real enterprise environment. Your example about Istio sidecars and ArgoCD pipelines is perfect - that's exactly where it fails.
I've run into this trying to simulate data pipeline evaluations. You can ask about "handling late-arriving data" and get textbook lambda architecture talk. A real data architect at a bank will immediately ask about idempotent upserts into a specific Delta Lake table pattern when their upstream Kafka cluster has exactly-once semantics but their legacy mainframe source doesn't. The model doesn't have the context to generate that specific, interconnected mess.
The workaround I've used is to build a detailed "context document" for each major persona and feed it in the prompt. It's tedious, but you can pre-load the scenario with your actual architecture diagrams, a list of their current tools, and even paste real past objections. Then the role-play gets closer to reality. Without that, you're just practicing for a conversation with a textbook, not an engineer.
—davidr
Yeah, the context document idea makes sense, but that sounds like a huge lift. How much detail are we talking? Like, do you have to manually update it every time a new tech stack or competitor comes up?
Totally get what you're saying about the generic concerns. It hits the same way in data pipelines. You can simulate a basic "how do you handle data quality?" question, but you miss the gritty follow-ups.
Like, they won't ask how your CDC tool handles a schema change in a source table *while* a backfill for that table is running, which is when everything breaks. That's the real test.
I'm curious, have you tried feeding it actual past sales call transcripts? I wonder if that would help it mimic the jump from generic to specific a bit better, or if it would still miss the hidden connections.
PipelinePadawan
The context document approach isn't a set-and-forget solution, it's basically building a proprietary, low-accuracy simulation engine. You have to manually update it constantly, and that's the real catch. Every new competitor feature, every weird architectural pattern from a major prospect, every time a new version of Kubernetes deprecates something critical to your value prop - it all needs a manual entry. It becomes a knowledge base maintenance job that likely costs more than the training value it provides. I've seen teams spend more time curating the "simulation" data than actually doing the role-play.
Trust but verify.
Exactly. Those follow-up questions are where the real evaluation happens. The generic simulation misses the entire integration puzzle you have to solve.
I've tried to force this by building scenario prompts that include a specific, messy tech stack fragment as immutable background. Something like: "You are evaluating this tool. Your environment: multi-cluster Istio, ArgoCD GitOps, and you've had issues with Envoy's memory spikes during high cardinality tracing. Start the conversation."
Even then, it often reverts to generic patterns after the first exchange. It can't hold the complex, interconnected state of a real platform in its "mind" for a full dialogue.
Ship fast, measure faster.
You're hitting on the core architectural limit. It's a context window and attention problem, not a prompt engineering one. The model can read your "messy tech stack fragment," but it can't genuinely *reason* about the interactions between Istio, ArgoCD, and high-cardinality tracing in a sustained way. It's pattern-matching on keywords, not simulating a system.
The reversion to generic patterns after an exchange is the tell. It's like it has a shallow cache for the scenario details that gets flushed by the next LLM inference step. I've seen this trying to simulate a POC troubleshooting session where the "prospect" needs to remember a faulty config they introduced three exchanges ago. The model always forgets.
So the workaround isn't a better prompt. It's accepting you're only getting a first-round draft of a conversation. The real training value is in forcing your sales engineers to identify where the simulation goes generic and *they* have to inject the specific, messy follow-up. That's the actual skill they need anyway.
You're absolutely right about the shallow cache. It can't hold onto the state of a complex, multi-component failure scenario.
I think that reversion to generic patterns is the most useful diagnostic, though. When a sales engineer in training spots that shift, it means they've hit the boundary of the simulation and need to switch from *reacting* to *driving* the conversation. That's the exact moment they'd need to pull out a whiteboard in a real call.
So maybe the value isn't in simulating the perfect prospect, but in creating a tool that reliably fails in this specific, identifiable way. It trains them to listen for the drop from specific to generic, which is a killer skill.
- GG
That's a great example. I've run into the same wall trying to simulate a complex POC for a Kubernetes cost tool.
You can get a generic question about resource recommendations, but it won't dig into the chaos of a real cluster, like asking how the tool's recommendations interact with a pending node pool upgrade or a misconfigured VPA. The model doesn't connect those separate issues.
Have you found any way to even slightly improve the depth, or is it always going to revert to the generic surface level after one question?
learning every day
That's the exact spot where I'd get stuck trying to simulate API evaluation calls. You get the generic "what about latency?" but never the sharp, specific follow-up.
It reminds me of webhook testing. You can prompt for "concerns about webhook reliability," but a real developer won't ask that. They'll ask: "How do you handle retries when our ingestion endpoint returns a 429 because of *your* burst of events, and does your retry logic respect our `Retry-After` header when we're at our rate limit?"
The model just can't thread those stateful, system-specific conditions together. It's all surface-level patterns.
Webhooks or bust.
That example you gave about the platform engineer's questions is perfect. It moves right past the textbook concerns and into the operational reality of integrating yet another tool into a live, complicated system. The "how does it handle" and "show me" questions are what separate a real evaluation from a scripted one.
I run into the same wall with project management platform demos. You can simulate a basic "how does task assignment work?" but you can't get it to role-play a product owner who's worried about how the new tool's sprint planning view will break their team's existing JIRA-to-slack notification workflow, because that workflow was a custom script their intern built two years ago. The simulation lacks that institutional memory.
It's not just about technical depth, it's about the unique, often undocumented, way a prospect's organization has glued their own stack together. The LLM can't simulate the tribal knowledge.
The right tool saves a thousand meetings.
That "tribal knowledge" point is spot on. It's the whole difference between a clean demo environment and the spaghetti junction of a real company's data pipelines.
We see this all the time with BI tool evaluations. You can practice answering questions about dashboard performance, but you can't simulate the person who's worried about how the new viz tool will break the 20 undocumented Crystal Reports exports that finance still runs every quarter, because their legacy system only accepts that specific output format. The model can't hold that kind of messy, inherited complexity.
Maybe the training value is exactly that - it forces you to anticipate those integration black holes before the call, because you know the simulation won't bring them up for you.
Data is the new oil - but it's usually crude.
Exactly. I've lived this with database migration tools. We built this whole "simulation" context about handling legacy MySQL schema quirks, and then PostgreSQL 14 dropped and changed how it handles certain index conversions. The document was immediately out of date.
It turns into a full time job just keeping the training material relevant, and you're always reacting, never proactive. The real learning happened when we abandoned the "perfect simulation" and just used the generic ChatGPT role-play to surface areas where our own internal knowledge was outdated. When it couldn't answer a prospect's question because our context docs were stale, that was the red flag for us, not for the trainee.
Backup first.
You're hitting the precise limitation of stateless pattern matching versus stateful system simulation. The questions a real platform engineer asks aren't isolated, they're interconnected probes into a specific operational reality.
This is the exact same problem we face in the database world when trying to simulate a migration evaluation. You can get generic questions about downtime, but you won't get the prospect who asks, "How does your online schema change tool handle a multi-terabyte table with a composite primary key and a pending `pt-online-schema-change` operation that was killed halfway, leaving triggers in place?" The model can't maintain the chain of dependencies, edge cases, and partial failures that define a real environment.
Your workaround of using the generic role-play to surface gaps in your own team's knowledge is the pragmatic path. It flips the script: the simulation's failure becomes a diagnostic tool for your internal preparedness, not a training tool for conversational depth.
SQL is not dead.
Your context document workaround is exactly where the vendor hype breaks down. You're not training a salesperson, you're just curating a knowledge base the model can weakly reference. That's a content management problem, not a training solution.
So you've built this meticulous document. How many hours to keep it current? And what happens when the prospect's "scar tissue" involves a system integration you haven't anticipated? The model will still default to its generic patterns the moment it steps outside your pre-loaded scenario. It's a brittle scaffold built over the same shallow foundation.
It feels like we're using the tool to automate the wrong part of the job.
cg