Alright, so we bit the bullet on Humata. The pitch was too good: drop hundreds of research papers (PDFs) into a bucket, ask it questions in plain English, get citations. Our old system was a nightmare of manual tagging in Notion and a prayer that someone remembered the right keyword. We thought we were automating the toil out of literature reviews.
Two months later, the on-call engineer is getting paged at 3 AM because a researcher is screaming about "hallucinated citations." Turns out, our shiny new AI assistant was confidently making up references to papers that didn't exist, or attributing findings to the wrong authors. The SLO for "accurate citation retrieval" was basically zero.
Here's the breakdown of what went wrong for our specific use-case:
* **The Black Box Problem:** You ask "Which papers discuss SLOs for machine learning models?" It gives you a list with snippets. But you can't *trace* why it picked those. No transparency into the ranking or retrieval logic. It's like a monitoring system with no query builder or logs.
* **No Fine-Grained Control:** With our manual system, we had defined tags like `#incident-response`, `#distributed-tracing`, `#cost-optimization`. You could combine them. Humata's "understanding" is a fuzzy match. It might bring up a paper about financial cost optimization when you asked about cloud cost optimization. The recall was high, precision was garbage.
* **The Confidence Deception:** It presents answers with the same assertive tone whether it's pulling from a rock-solid source or the abstract of a vaguely related paper. No confidence scores, no "hey, I'm not sure about this one." As an SRE, if my alerting system did that, I'd rip it out immediately.
We're going back to a *semi*-manual system, but with better tooling. We're using a simple script to extract text and metadata, feeding into a proper vector store (pgvector) we can actually query and tune.
```bash
# Something like this is now our ingestion pipeline
for pdf in *.pdf; do
text=$(pdftotext "$pdf" -)
metadata=$(extract_metadata "$pdf") # custom script
# Embed and store in our own DB, not a black box
store_embedding "$text" "$metadata"
done
```
Now we can build our own retrieval, set our own similarity thresholds, and actually explain why a result was returned. It's more work upfront, but the mean time to repair (MTTR) for a research query has gone down because we trust the results.
Humata felt like replacing a custom dashboard with a magic AI chart that sometimes shows you data from the wrong cluster. Great for a casual overview, fatal for incident response. Or, in this case, academic research.
-shift
Pager duty is not a hobby