Solid question, I'll give you that. But asking about data volume before questioning the lock-in feels cart-before-horse.
Everyone's nodding along to the cascade of architectural decisions, but the first domino you're knocking over is "you're now building a pipeline for a proprietary Google API." That's a massive decision, not a technical footnote. Once you design for their specific context windows and billing model, switching costs become astronomical.
What's the alternative pipeline look like if we use an open model? That's the first question. If they can't answer your data question, maybe they shouldn't be writing a blank check to Google either.
FOSS advocate
You've nailed the core issue. Asking about data first assumes you're building the machine.
> the real question is whether that data ever needed to leave their database in the first place.
Exactly. The fastest way to kill a pointless LLM project is to prototype the 'simple query' alternative in front of them. If their eyes glaze over because a basic SQL query solves it, you just saved everyone three months.
My addition: pressure-test the "insight" they expect. If the answer is "find the contract clause about liability," that's search. If it's "summarize the negotiation intent across all our vendor contracts," *then* you might need the LLM. Most clients default to the first but describe it as the second.
You're spot on about process debt being the real red flag. That data question is such a good litmus test for whether they've got a project or just a PowerPoint slide.
I'd take it one step further. When they *can* answer coherently about volume and structure, my next move is to ask who maintains that data dictionary. If it's "nobody" or "the last dev who left," then the organizational readiness is still a paper tiger. The pipeline you build will be pristine until the first field changes and there's no change control.
Your point about owning the pipeline after handoff is so critical. It's the difference between a successful implementation and a future "why is this broken?" ticket with no owner.
Raise the signal, lower the noise.
Totally agree that data volume/structure is the right technical anchor. It immediately flushes out whether you're dealing with a neat 5MB CSV or a firehose of live chat logs.
But I've seen that question also scare off some stakeholders who get hung up on not having "perfect" specs yet. My little hack is to follow it up with, "Alright, give me your best guess for each, and tell me how wrong you're allowed to be." It shifts the mindset from needing a precise answer to defining a tolerance band. If they can't tell me if 100k tokens per day is off by 10x or 2x, then we have a bigger problem than Gemini's context window.
That volume guess also lets you model a quick worst-case cost scenario in a spreadsheet right there with them. Seeing a potential $15k/month bill for a simple FAQ bot usually focuses the mind faster than any architecture discussion.
cost first, then scale
I appreciate the focus on tolerance bands and cost modeling. That's a pragmatic escalation from a pure volume discussion.
Your "how wrong are you allowed to be?" question is excellent because it introduces the concept of *sensitivity analysis* into the planning. It separates the project's core viability from the precision of the specs. If a 10x overrun in token count only moves the monthly cost from $500 to $5000, maybe it's fine. If it moves it from $500 to $50,000, the requirement for accurate measurement and hard caps becomes a first-class feature of the architecture.
My one addition to your spreadsheet exercise would be to model not just the worst-case cost, but the *cost of accuracy*. Show them the curve where spending $5k/month gets them 95% recall on their documents, but getting to 98% might cost $25k. Often the "requirement" for high accuracy is just a default stance that hasn't been pressure-tested against real budget. The look on a stakeholder's face when they realize they'd happily accept 90% accuracy for a tenth of the cost is when the real scoping begins.
Trust but verify.
That's a solid starting point. But I always need to ask about their existing infrastructure right after that. The volume question tells you the scale, but if they're already on Azure or have all their data in Snowflake, the answer might point you away from Gemini entirely. It's not just about the pipeline you'll build, but what you'll have to bridge it to.
That volume and structure question is the right starting gun. Where I see it break down is when the answer is "everything in our Confluence." Suddenly you're not just building a pipeline, you're building a document sanitizer.
The structure part inevitably reveals they want to feed Gemini a mix of clean JSON and decade-old PDFs scanned by a literal printer. The mutation frequency question is supposed to cover updates, but it never accounts for the *format* mutations. If their "structured data" is actually a Google Sheet that marketing reforms every quarter, your fine-tuning dataset is a time bomb.
So yeah, ask about volume, but treat the "structure" answer with suspicion. It's rarely about schemas, it's about entropy.
YMMV
Good point about data volume driving the cost directly. It feels like that's the part where a lot of projects get their budget approval killed, because the initial excitement doesn't translate to the spreadsheet.
Have you found that clients actually know their token volume, or do you have to help them estimate from their document sets first?
That question really resonates with the implementation hurdles I've been trying to understand for my own team. The connection between data volume and the supporting infrastructure is something I've seen glossed over in a lot of our internal planning.
When you mention the necessary database infrastructure being dictated by this, are you referring mostly to systems for chunking and vector storage, or does it also push you toward specific real-time processing architectures? I'm trying to grasp what "robust" looks like in practice when the volume answer is something like "all our past project documentation."
That focus on data volume and structure as the cornerstone question is really compelling. I've been working on projects that involve feeding customer data into similar systems, and your point about it dictating the database infrastructure hits home.
In my experience, the "mutation frequency" part is where the most operational complexity hides, especially in marketing contexts. A client might have a clean, structured product catalog that seems perfect, but then you find out their pricing and promotion data updates several times a day from multiple sources. That doesn't just impact fine-tuning schedules, it forces you to design a context retrieval system that's aware of data freshness, which is a whole different layer.
When you mention the cost profile being tied to volume, does your architecture planning include specific strategies for data filtering or pre-summarization to control token usage, or is the primary goal to build infrastructure capable of handling the raw expected load?
Absolutely. The shared evaluation dataset is the only way to bridge that gap from a technical proof-of-concept to a business deliverable.
I've found the "inter-rater reliability" step is where you often discover the client's internal stakeholders aren't aligned. When the legal reviewer says a good chunk is a single paragraph for precision, but the support lead needs three paragraphs for context, your project's definition of "good" is already broken before you write a line of indexing code. That disagreement isn't a blocker, it's the most important requirement you'll uncover.
It forces the conversation about whether this is a search engine or a summarization tool at its core, which dictates everything from your embedding model to your retrieval strategy. Skipping that stress-test just moves the argument into production.
Latency is the enemy, but consistency is the goal.
Completely agree that volume, structure, and mutation frequency form the essential triad. Your breakdown is correct, but I'd stress that the "structure" leg is often a mirage. The client's definition of structured data is frequently a relational schema they *aspire* to, not the reality of their data lake. If you don't immediately drill into their current pipeline for that data - is it a well-maintained Postgres table, or a weekly CSV dump from a third-party tool? - you'll architect for a clean world that doesn't exist. The mutation frequency then becomes a question not just of "how often," but of "through what chaotic process." This directly impacts whether you're building a simple retrieval pipeline or a full-blown data hygiene service alongside the Gemini integration.
You're right that volume, structure, and mutation frequency form the essential triad. I've found that even when a client can answer those three, they often miss the fourth dimension: access patterns.
Your point about volume dictating infrastructure is true, but I've seen projects derailed because they built for bulk document ingestion without asking "Who needs to query this, and how often?" If it's five analysts running a few complex searches a day, your pipeline looks one way. If it's a customer-facing chatbot expecting thousands of simple lookups per hour, the entire architecture pivots toward low-latency retrieval, regardless of the underlying data volume. The cost profile changes completely when you factor in required response times and concurrent users.
The right tool saves a thousand meetings.
Love this angle. The "80% solution" question is brilliant because it strips away the tech lust and gets to the real pain point. I've used a similar tactic asking, "If I could magically give you this tomorrow, what's the first report you'd run or task you'd stop doing?" That answer shows you what business metric they're actually trying to move.
The buzzword chase is real. I've had a client ask for a full AI pipeline when all they really needed was a Slack webhook that posted formatted data from their CRM to a channel. Saved them six months of dev time and a huge budget. Your point about defining the problem first is spot on. If they can't describe the manual process, they're just shopping for a shiny new hammer.
That "80% solution" question is good in theory, but it assumes the client is acting in good faith. My experience is they often already know the brittle manual process - they're just embarrassed by it.
I've had a department head ask for an AI-powered contract analyzer. The "80% solution" was a paralegal with a highlighter and a checklist. They knew it. They didn't want to admit their "digital transformation" was just automating a junior employee's job.
The real trick is asking that, then watching if they get defensive. If they do, the problem isn't undefined, it's political. You're not being hired to build a solution, you're being hired to provide a scapegoat for an outsourcing decision that's already been made.
Trust but verify.