Alright, let's get this out there before the usual chorus of "just use the SaaS, it's easier!" drowns out any rational thought. I've been watching the ChatPDF hype train leave the station for a while now, with everyone happily tossing their internal documents, financials, and who-knows-what-else into some third-party's vector database. The pricing tiers are a classic "land and expand" play, and the privacy policy reads like a work of speculative fiction.
So, I did what any self-respecting infra skeptic would do: I spent a weekend building my own. The goal wasn't to replicate ChatPDF feature-for-feature with a slick UI. The goal was to answer one question: "Can I get 80% of the utility for 20% of the cost and 0% of the data exfiltration risk?" The answer, predictably, is yes—if you're willing to get your hands dirty.
Here's the gist of my setup, living entirely within my own AWS account (and could just as easily be on-prem):
* **Ingestion & Chunking:** A simple Python Lambda (triggered by S3 uploads) using `PyPDF2` and `langchain`. No magic here, just basic text splitting.
* **Embeddings:** This is where you have to make a choice. I'm using `sentence-transformers` (all-MiniLM-L6-v2) running in a separate Lambda with a bit more memory. Yes, it's slower than OpenAI's API, and yes, the vectors might be marginally less "good." For internal technical docs and RFCs? It's perfectly adequate. The cost is zero beyond the Lambda runtime.
* **Vector Store:** PostgreSQL with the `pgvector` extension on a small RDS instance. I looked at Pinecone and Weaviate, but adding another external service defeats the purpose. This keeps everything under my existing backup/security umbrella.
* **LLM:** This is the big one. I'm running `Llama 3.1` (8B) via `ollama` on a small, cheap G5 instance (Spot, of course). The model is loaded into GPU memory, and it sits there waiting for queries. The prompt template is simple: "Use the following context to answer the question..."
* **Orchestration:** A simple FastAPI app on ECS Fargate that ties it all together: takes a question, fetches relevant chunks from PG, stuffs them into the prompt, sends it to the local LLM, streams back the response.
The workflow isn't as seamless as a drag-and-drop web interface. I have to `curl` my API or use a janky Streamlit frontend I slapped together. But let's talk brass tacks.
**Cost Comparison (Monthly, Approx.):**
* **ChatPDF Pro Plan:** $20/user/month. For a team of 10, that's $200. And your data is... somewhere.
* **My Setup:**
* RDS db.t3.small (~$25)
* G5 Spot Instance (~$45)
* S3, Lambda, Fargate (~$15)
* **Total: ~$85** for *unlimited* documents and users, with predictable scaling costs.
**The trade-offs are real, and you should stare at them:**
```python
# This isn't always perfect. You will get hallucinations.
# You must implement a "I don't know" fallback based on context relevance score.
if max_similarity_score < 0.7:
return "The provided documents do not contain a clear answer to this question."
```
It's more operational overhead. I have to monitor the instance, manage deployments, and handle my own chunking logic. But I sleep better knowing that the sensitive architecture diagram I fed it isn't part of some training run or sitting in a multi-tenant database with who-knows-what access controls.
Is this for everyone? Absolutely not. Most people should probably just pay the toll. But if you have the infra skills, care about your data's lineage, and have a healthy distrust for monthly subscriptions that creep ever upward, rolling your own is a viable and deeply satisfying middle finger to the entire "just upload it" paradigm.
-- cynical ops
Your k8s cluster is 40% idle.