Just built a custom knowledge base chatbot with Le Chat and Next.js - here's the repo.

So you're worried about cost and accuracy. Good, you should be.

The real trap here is thinking of this as a one-time integration cost. Le Chat's API pricing seems fine until you actually have users. That "straightforward" embedding step? That's a bill per document, per chunk. Streaming responses? You pay by the token as they generate, and you can't cap it if someone asks a question that pulls in a huge context.

Best practice is to treat every API call like a meter that's running. You need to aggressively cache embeddings, chunk your documents efficiently to minimize calls, and set hard limits on context window size. Otherwise, your neat chatbot becomes a money pit.

Has anyone built something similar? Sure, lots of us have. And then we watched the bill.

That's a good point about treating calls like a running meter. From an accounting perspective, you'd want to track that as a variable cost per user session. Have you found a good way to estimate that cost upfront? Like, a rough cost per question based on average document size?

Oh, you found the context challenge "straightforward," did you? That's the red flag.

Your "main challenge" is the entire cost equation. Every time someone uploads a fresh PDF, you're running embeddings across every chunk. That's not a one-time dev challenge, that's a recurring, unbounded line item. And structuring the system prompt for accuracy? That usually means shoving *more* context into the window, which just makes each chat completion more expensive.

You can't talk about best practices for cost and accuracy separately. They're the same problem. Chunk too small for accuracy? Costs go up because you need more chunks per answer. Chunk too big for cost control? Accuracy tanks because the model gets lost. You built a prototype. Now go calculate how much it costs to answer a single question from a 50-page PDF. I bet you won't like the number.

Straightforward integration isn't the win you think it is. The real trouble starts when you move from prototype to production and that "simple" embedding call runs on every new user upload. Your bill scales with user activity, not just your own testing.

You're asking about system prompt structure for accuracy, but that's how you blow the context window and costs. Every extra token for "accuracy" is another line item. Le Chat doesn't care if your answer is correct, just that you used the tokens.

Post the repo. Let's see how you're chunking. That's where the real cost is hidden.

You're right that embedding cost scales with uploads, not just dev work. But there's a middle ground between "ignore it" and "it's a trap."

Caching embeddings is the obvious start, but you also need a cleanup strategy. Old, unused documents still cost you storage on their platform, which isn't free. A production system needs to tie document retention to user activity, or you're paying for stale data.

The more subtle point is about chunking. If your chunking strategy is naive, you're not just paying for more initial embeddings. You're also forcing the retrieval step to pull more chunks per query to get the same answer, which multiplies the context window size and the completion cost. It's a double tax.

So yes, post the repo. The chunking logic is the first place to look for those multiplicative costs.

Exactly on the chunking double tax. It's the multiplier everyone misses. A bad split doesn't just mean a bigger initial embedding bill. It makes retrieval sloppy, so you end up pulling four low-relevance chunks instead of one precise one, ballooning the context window for every single query.

That cleanup strategy point is huge, too. It's not just about storage fees on their end. If you don't prune unused data, your vector search gets noisier over time. Retrieving from a million stale chunks is slower and less accurate than from a clean, active set, which again, hits your token usage.

Your chunking logic is your unit economics. I'd love to see how they handle overlap and boundary detection. A naive character split can slice a critical table or definition right in half, guaranteeing you'll need to fetch both surrounding chunks to make sense of it. That's the tax.

You nailed it with the retrieval sloppiness. That's where the cost balloon really happens.

I'd add that the "clean, active set" is harder than it sounds. If you prune based purely on last access time, you might axe reference docs that are critical but rarely pulled - like a company policy PDF. You need a smarter heuristic, maybe weighting chunk relevance scores over time.

And on chunk boundaries - yeah, splitting a table is a classic blunder. Semantic chunking that respects structure is key, but then you're trading off that initial simplicity OP mentioned. It's never just a character count split in production.

Good point about weighted relevance scores over time. You'd need to log retrieval relevance for each chunk per query, not just last access. That log itself becomes an operational burden though. You're adding storage and compute just to make your pruning decision.

Semantic chunking that respects structure is correct, but it pushes complexity upstream. Now your document ingestion pipeline needs to understand PDFs, markdown, HTML, or DOCX structure. You're not just building a chatbot anymore, you're building a document processing service with its own failure modes.

That's the real trade-off everyone dances around. The cost isn't just API tokens. It's the operational overhead of managing the knowledge base itself.

Straightforward integration means they handled the complexity for you, but you're still on the hook for their pricing model. You're building on quicksand.

You think your main challenge is prompt structure. It's not. It's that your cost per query is now tied to the quality of that prompt. Every tweak for "accuracy" adds tokens, and tokens are the product they're selling. You're not optimizing a system, you're filling a meter.

Post the repo. Let's see if you cached embeddings at all, or if you're just blindly calling their API on every refresh. That's the first bill shock.

Glad you got streaming working, that's a win. On system prompts, I've found less is more. A tight prompt focused on "answer only from the provided context" reduces hallucination and keeps token counts down.

The repo looks good for a first pass. For the cost side, I'd add a check to see if an embedding for a file already exists before calling the API. It's a small change that prevents re-running embeddings on the same PDF across user sessions.

Has anyone benchmarked different chunking strategies against Le Chat's pricing? I'm curious if overlapping chunks are worth the extra initial cost for the retrieval improvement.

I agree that a simple embedding cache is a necessary first step, but it's a partial solution. The cache key must be deterministic, typically a hash of the file content plus your chunking parameters. If you change your chunk size or overlap later, you invalidate the entire cache, which is a hidden migration cost.

Regarding benchmarking chunking strategies against Le Chat's pricing, I haven't seen published data. The value of overlapping chunks isn't just retrieval improvement; it's a hedge against poor boundary placement. A 10% overlap might increase your initial embedding cost by 10%, but if it prevents a 30% increase in retrieved chunks per query due to fractured context, the math works out quickly. You'd need to instrument your retrieval to track the average number of chunks per query and their relevance scores with and without overlap.

Hedging with overlap makes sense, but it's a static fix. If your document structure changes or you add new file types, your fixed 10% overlap might still cut a table in half. You need to pair it with at least basic structure detection for common formats.

Instrumenting retrieval is the real work. Without tracking average relevance score per chunk per query, you're just guessing on the overlap math. I log chunk IDs and their cosine similarity scores for every query. It's extra storage, but now I can prune based on actual usage, not assumptions.

The cache key point is critical. Hashing content + chunk params saved us from a massive surprise bill after we tweaked the overlap.