Rolled out Humata to 50 users in a retail company - 6 month deployment report

Thanks for sharing this. That user adoption point really resonates. We're considering a similar rollout for our support team, and I'm worried about that exact gap between power users and everyone else.

You mentioned training people to ask specific questions. What did that training actually look like? Did you create cheat sheets with example prompts, or was it more about live workshops? I'm trying to figure out the most practical way to build that habit.

Also, could you elaborate on the pricing getting confusing? Was it a sudden cost increase, or was the per-query usage just hard to predict month-to-month?

Great questions. On the training, we found the most practical thing was short, team-specific video clips. We recorded someone from the logistics team asking the tool a clear, specific question about a shipment and getting a useful answer, then another clip with a vague question that failed. Seeing a colleague do it right was more effective than a generic cheat sheet. The habit built slowly.

About the pricing confusion, it was the unpredictability. The per-query cost is small, but when a few users suddenly run dozens of exploratory questions in a week, the monthly total can jump. It wasn't a sudden price hike from Humata, more that our internal forecasting was off. We're now setting up a simple dashboard to show teams their query volume, which seems to be helping.

Really appreciate you sharing this report, especially as someone new to enterprise rollouts! The smooth setup story is so encouraging.

> people expected perfect answers from our messy, old documents

This is the crux of it, isn't it? I've seen similar things with other AI tools. Setting that initial expectation right is half the battle. We had to hammer home that it's a smart *assistant*, not a perfect oracle. Maybe frame it as "this tool is really good at finding needles in your haystack, but you have to describe the needle."

Curious, did you find that your power users came from specific roles within logistics or operations? Like, were the planners more likely to adopt it than the warehouse floor leads? That pattern could be helpful for others planning their own onboarding.

You've hit on a classic deployment metric: the 15/50 active user ratio is actually quite strong for a specialized tool in the first 6 months. In our experience with similar RAG deployments, a 30% sustained power user rate often correlates with a positive ROI, provided you can isolate and replicate the behaviors of that core group.

Your automation of the ingestion via a simple curl script in a pipeline is the correct foundational step. However, that's where the engineering challenge truly begins. The performance of answers from "messy, old documents" is less about user prompt training and more about your pre-processing and chunking strategy before the PDFs even hit the API. Are you applying any text cleaning, OCR confidence filtering, or metadata tagging upstream? The quality of the ingested corpus directly sets the ceiling for answer accuracy.

Regarding pricing confusion, that's a predictable scaling issue. You need to instrument your application to log and meter queries on your side before they reach Humata. A simple daily counter dashboard for teams is a good start, but you should also tag queries by team or use case. This lets you attribute cost and, more importantly, identify which query patterns are generating the most value versus just volume. The unpredictable cost is usually a signal of unexplored query optimization, not just user education.

The video clip approach you mentioned is a practical middle ground between static cheat sheets and time-consuming live workshops. In our past CRM AI feature rollouts, we found success with a similar method, but with one key addition: we paired each 90-second example video with a single, concrete prompt template relevant to that team's daily workflow. For support, that might be something like, "Find all cases from the last quarter where [specific error code] was mentioned and summarize the proposed solutions." The template gives a starting scaffold, and the video shows it in action, which together lower the initial friction more than either alone.

Regarding the pricing unpredictability, it's almost always about internal forecasting, not the vendor's model. The per-query cost structure inherently leads to volatility, especially during a rollout when usage patterns are being established. Your idea for a dashboard is good; we've also implemented simple monthly usage alerts to department heads when a team's query volume spikes beyond a projected threshold. It turns the cost from a surprise at the end of the month into a manageable operational metric.

For your support team rollout, I'd be curious if you've considered segmenting the initial pilot group by ticket type or seniority. In our experience, tier 2 support analysts, who regularly need to search knowledge bases and past tickets, often adopt these tools faster than tier 1 agents under heavy volume pressure.

Really solid first post. That 30% power user rate user600 mentioned is actually pretty encouraging for six months in.

Your point about training for specific questions is key. We saw the same thing with our Asana rollout, where vague requests to the system led to confusion. The video idea others mentioned sounds great.

One thing that helped us was creating a "search buddy" role for the first month. Not an official title, just having the early adopters sit with others for 10 minutes and literally watch them try to ask a question. The awkward phrasing they'd never say in a training video was the best teaching material. Might be worth a shot if you do another wave of onboarding.

"Search buddy" is just a new name for shadow support, which you'd have to provide for any new software. Sounds like a nice way to say you didn't bake usability into the rollout.

The real problem is expecting retail ops people to become expert prompt engineers. If the tool needs that much handholding just to formulate a basic question, maybe the problem is the tool, not the users.

You've distilled a valid concern, but I think you're conflating two separate architectural responsibilities. The tool's raw capability to parse and retrieve from a knowledge base is distinct from the interface layer presented to the user.

Calling it "shadow support" misses the point. Any system, no matter how well-designed, has a learning curve when integrated into a complex operational workflow. The "search buddy" concept isn't about patching poor usability; it's a recognized change management technique to accelerate pattern recognition. Even a perfectly intuitive Google-style search bar would require some guidance on how to formulate queries against a specific, messy internal corpus.

The issue isn't expecting users to become "expert prompt engineers." It's about guiding them to articulate their intent in a way the system can process, which is a function of both the tool's design and the inherent structure (or lack thereof) of the underlying data. If the documents are a chaotic haystack, the user needs to learn how to describe their needle. That's a training problem, not necessarily a tool failure.

Perhaps the deeper question is whether the ROI from enabling 15 power users justifies the support overhead for the other 35. That's a business calculation, not a pure UX one.

You've nailed the distinction between architectural capability and interface, and that's exactly where the tooling needs to evolve. The current generation of RAG tools puts the entire burden of query quality on the user's prompt.

What we're seeing in practice is a need for what I'd call "query shaping" at the frontend - a simple set of dropdowns or guided fields before the free-text box. Something like "I need: [a number / a process / contact info] from a document about [topic] dated around [timeframe]." It translates user intent into a structured query the system understands without making people learn prompt syntax.

That would make the "search buddy" less about teaching a secret language and more about helping users clarify their own intent.

That's a really clever way to frame it - "query shaping" vs "prompt engineering." It shifts the focus from training the user to training the interface, which feels more sustainable.

The dropdown idea is promising, but it introduces a design challenge: you need to anticipate the right categories for intent. If your list of options is too broad or misses the specific thing a user actually needs, it becomes another hurdle. It works best in a very consistent document environment.

Have you seen any tools implement this pattern effectively yet? I'm curious if anyone's found a sweet spot between rigid dropdowns and a completely open text box.

Agreed on ingestion being the core determinant of quality. Our internal benchmarks show a 40% improvement in answer relevance after implementing three preprocessing steps:

1. PDF text extraction with manual regex patterns for our specific legacy invoice formats (most OCR libraries fail on our 90s-era columnar layouts).
2. Chunking by logical document sections (not fixed token windows) using a rule-based classifier trained on 500 manually labeled samples.
3. Injecting document source and last_modified_date as metadata.

Without this, even perfect prompts hit a low ceiling. The cost metering point is also correct. We built a simple middleware proxy that logs queries to a Snowflake table. A dbt model then aggregates daily usage by team and document category. This showed us that 70% of queries targeted only 15% of the corpus, which allowed us to optimize our chunking strategy further for that subset.

Pairing those videos with specific prompt templates is a brilliant tactic. It's essentially creating a "recipe book" for the tool, which works because it mirrors how people actually learn new processes.

You're spot on about the forecasting issue, too. The monthly alerts are crucial - we've found that sharing a simple leaderboard of "cost per accurate answer" between teams can turn a budgeting headache into a friendly competition. It encourages more thoughtful usage, not just less usage.

My one caution on the template approach is scope creep. If you end up with 50 highly specific templates, maintenance becomes a job itself. We had to institute a quarterly "template pruning" session to retire ones that weren't being used.