Skip to content
Notifications
Clear all

Just built a dashboard to track our team's Humata usage and cost per query.

19 Posts
19 Users
0 Reactions
198 Views
(@cloud_cost_hawk_new)
Reputable Member
Joined: 7 months ago
Posts: 333
Topic starter   [#25729]

We've been using Humata for a few months now, mostly for querying technical docs and architecture diagrams. The initial "per user, per month" price seemed straightforward, which should have been my first red flag. Once the team got hooked on asking it everything, our usage exploded and the bill followed. Surprise, surprise.

I got tired of the opaque "number of questions" metric from their dashboard, so I built a scrappy internal tool to track actual usage and, more importantly, cost per query. The results are... enlightening, if not outright depressing.

Here's the gist of what we're pulling and calculating:
* **Raw Usage:** We pull daily question counts per user via their API (when it feels like working).
* **True Cost:** We factor in our team's specific tier (Pro, with some annual commitment nonsense).
* **Cost per Query:** (Monthly Subscription Cost / Total Team Queries). Simple, but they don't show it.

A sample from our logging script (names changed):

```python
# Pseudo-output from our daily cron job
Date: 2024-10-28
User: dev_engineer_1
Questions: 147
Estimated Cost (prorated): $4.12
Cost per Query: ~$0.028

User: product_manager_1
Questions: 23
Estimated Cost (prorated): $0.64
Cost per Query: ~$0.028
```
See the problem? The quiet user subsidizing the power user, all while the per-query cost only makes sense if you're hitting a high volume. For low-volume teams, this model is brutal.

Key takeaways so far:
* The "unlimited" questions tier is a myth unless you have infinite users on the plan.
* Vendor lock-in is already setting inβ€”all our documents are in their system, and the query history is a form of training data we can't easily export.
* Without tracking, you're flying blind on whether you're getting value or just burning cash on "what does this diagram say?" queries that a simple Ctrl+F could solve.

I'm now pushing for a hard monthly query limit per user before we need to re-evaluate. Has anyone else done a similar breakdown? I'm curious if the Enterprise tier's "custom pricing" actually breaks down to a sane per-query model or if it's just the same headache with a bigger commitment.

-- cost first


-- cost first


   
Quote
(@ethanb8)
Reputable Member
Joined: 4 months ago
Posts: 417
 

That's a clever way to get visibility into the real unit economics. The "cost per query" metric is something more SaaS vendors should surface, especially with consumption-based or usage-heavy tiers.

I'm curious, though, how are you handling shared or team-level credits in your calculation? For instance, if there's a pooled quota of "questions" that gets refreshed monthly, a high-volume user one week could make the cost per query look great for a light user later in the cycle. Does your model spread the flat monthly cost across all queries evenly, or do you attribute cost more directly to the user who consumed the credit at the time?

It's that kind of granularity that often gets lost, even in homegrown trackers.


Keep it civil, keep it real


   
ReplyQuote
(@chrisr)
Reputable Member
Joined: 4 months ago
Posts: 227
 

You've hit on the crucial accounting distinction between averaging and attribution. In our model, we treat the monthly subscription as a sunk cost, a fixed pool of credits. The "cost per query" is simply the total monthly cost divided by the total monthly queries, providing a blunt, team-wide efficiency metric.

However, we also run a separate attribution layer, which is where the real insights for platform management appear. We tag each query by user and project code. This lets us see which users or projects are depleting the shared pool most rapidly. While the financial cost is averaged, the operational cost--the risk of exhausting credits mid-cycle and degrading service for everyone--is directly attributable. That attribution is what we use for internal chargeback and to guide behavior, not the averaged financial figure.

Averaging gives you the macro unit economics, but attribution shows you the source of the variance. You need both to manage a shared resource effectively.


Data over dogma


   
ReplyQuote
(@chrisw2)
Reputable Member
Joined: 4 months ago
Posts: 309
 

Exactly - the pooled credits model completely warps the unit cost. We saw the same thing.

Our dashboard shows two numbers side-by-side for that reason:
* **Avg. Cost per Query:** Total monthly spend / total queries. This is the financial efficiency number.
* **Attributed Cost per User:** This uses a simple FIFO logic on the credit pool. If userA burns 100 credits on day one, they're charged against the first, most expensive credits in the pool. It's not perfect, but it stops high-volume users from looking artificially cheap.

Without that second view, you can't tell who's actually driving the burn rate.


Run it yourself.


   
ReplyQuote
(@benchmark_nerd_1337)
Prominent Member
Joined: 6 months ago
Posts: 547
 

The FIFO attribution method is a pragmatic start, but it's inherently lossy for cost allocation. You're still assigning an arbitrary sequence to a fungible pool. A more rigorous approach for behavior analysis is to calculate the marginal cost per query at the moment of consumption, which requires knowing the tier's credit quota and the burn rate.

For instance, if your Pro tier has 5000 monthly credits and costs $500, the first query's marginal cost is near zero (you've paid for the pool). The 5001st query's marginal cost is effectively infinite, as it may force an upgrade or overage. Tracking the query count against the remaining credit pool at the time of execution gives you a dynamic "risk cost" that FIFO can't capture.

This is why we log timestamps and compute a running utilization percentage. A query executed at 90% pool consumption carries a much higher operational cost than one at 10%, regardless of user.


numbers don't lie


   
ReplyQuote
(@danielr)
Reputable Member
Joined: 4 months ago
Posts: 408
 

Building your own dashboard to track this is a necessary hack, but it's a symptom of the real problem. The vendor's entire pricing model is designed to make unit economics opaque. You shouldn't need a logging script to figure out what you're actually paying per question.

That "straightforward per user, per month" price is a classic trap. It creates a false sense of predictable cost, while the variable consumption component, the "questions," is what actually drives the bill. You're now managing a shadow accounting system just to understand your own spend.

The depressing results you're seeing are the point. The next logical step is realizing you're spending engineering time to audit a vendor who should be transparent by default. At what point does the cost of building and maintaining your tracking tools outweigh just finding a service with clearer pricing?


Trust but verify.


   
ReplyQuote
(@briang)
Estimable Member
Joined: 4 months ago
Posts: 119
 

That's a good point about the hidden maintenance cost. We're already thinking about how to automate the data pulls because the manual effort adds up.

But doesn't that just prove the vendor's model is broken? If the cost of monitoring the service becomes a project itself, you're right, it defeats the purpose. I guess the question is whether the insights from the dashboard are valuable enough to justify that ongoing cost, or if it's just a stopgap.



   
ReplyQuote
(@cloud_ops_learner_2)
Honorable Member
Joined: 5 months ago
Posts: 561
 

I've been down this exact road with other "simple" SaaS pricing models. That cost per query you calculated, the ~$0.028, is the kind of number that makes you stop and think. When you see it in isolation, it seems trivial, right? But multiplied across a team, it adds up fast.

Your logging script is a great start. One thing we added was tagging the query *type* (e.g., "doc_search" vs "code_explanation") to our internal tracker. It helped us spot when people were using it as a glorified Ctrl+F instead of for complex analysis, which was a big driver of low-value, high-volume queries.

It's a bit of a bummer that we have to build these tools just to understand our own spend. But honestly, seeing those numbers in a dashboard did more to curb frivolous usage than any internal memo ever could.


Infrastructure as code is the only way


   
ReplyQuote
(@cloud_infra_vet)
Honorable Member
Joined: 5 months ago
Posts: 389
 

The prorated cost per query is the most actionable metric you've surfaced. I've found that putting a tangible dollar amount next to a seemingly free action creates immediate behavioral change. Your $0.028 figure is a perfect example.

However, that number's accuracy depends entirely on the stability of your data pipeline. You mention the API "when it feels like working." If your daily cron job misses data, your cost per query becomes artificially low, because you're dividing a fixed cost by an undercounted denominator. You need to build in reconciliation against the vendor's final monthly invoice, otherwise you're making decisions with flawed data.

We automated this by storing a daily snapshot in S3 and running a weekly diff against a manually uploaded CSV of the official bill. The discrepancy was often over 10%, which completely changed the narrative around which teams were the most expensive.



   
ReplyQuote
 amym
(@amym)
Trusted Member
Joined: 4 months ago
Posts: 85
 

So you're basically estimating the cost by prorating the flat monthly fee across all queries that month. I've been trying to think about how to do this for our team, and I keep getting stuck on a basic question: how do you handle queries from people who joined or left mid-month? If someone joins on the 20th and runs a bunch of queries, do you still divide the full monthly cost by all queries, including theirs? It feels like that would unfairly load cost onto the queries from the rest of the team who were there the whole time. Or do you prorate the user's subscription cost first before folding it into the calculation?



   
ReplyQuote
(@annac)
Reputable Member
Joined: 4 months ago
Posts: 391
 

Great question. That's a nuance we had to figure out, too. We handle it by using a daily prorated cost for each user.

So for a user who joins on the 20th of a 30-day month, we only assign 1/3 of the monthly seat cost to them before we factor their queries into the pool. It means the "full month" users' queries carry a slightly higher cost per query until the new person's pro-rated fee gets added in. It's not perfect, but it feels fairer than loading their queries onto the existing team's cost.

We use a simple rule: if a user is active for less than 75% of the billing cycle, they get prorated.


Keep it simple.


   
ReplyQuote
(@bobw)
Reputable Member
Joined: 4 months ago
Posts: 342
 

The 75% rule is a really smart way to simplify the pro-ration logic without getting lost in the weeds. I'd just add a small warning from our own setup: watch out for mid-month *role changes* that affect seat cost. If someone switches from a Pro to a Basic plan on the 15th, you need to apply that daily prorate to both cost segments before merging them into the pool. We missed that at first and our 'cost per user' graph went a bit haywire until we added a check for subscription tier history.

It's all about making those attribution rules slightly more granular than the vendor's billing system, isn't it? 😅


null


   
ReplyQuote
(@ethanp)
Reputable Member
Joined: 4 months ago
Posts: 371
 

You've zeroed in on the core frustration with that initial "straightforward" price signal. The moment you have to build an internal tool just to derive a basic metric like cost per query, the vendor's transparency model has failed. Your method of prorating the subscription cost across total queries is the logical starting point, but as the thread shows, it quickly opens a Pandora's box of attribution complexities.

The sample data is telling. A 6x difference in query volume between a developer and a product manager on the same day isn't inherently bad, but without that cost-per-query lens, you'd never see the disparity in resource consumption. That visibility alone can drive more conscious usage, but it's work the vendor should be doing for you.

The real question your dashboard raises is one of accountability. When you present these figures to your team, are you implicitly accepting responsibility for managing a cost model the vendor designed to be opaque?


Let's keep it constructive


   
ReplyQuote
(@davidn3)
Reputable Member
Joined: 4 months ago
Posts: 277
 

Your 75% rule is a practical cutoff for automation, but it introduces a small accounting distortion. What about a user active for exactly 75% of the cycle? Under your rule, they'd bear the full monthly cost, but they consumed only three-quarters of the time-based resource.

We sidestepped this by always calculating a daily prorated cost, regardless of tenure. The formula is straightforward in the data pipeline:
```sql
daily_user_cost = monthly_seat_cost / days_in_billing_month
```
Then we aggregate the `daily_user_cost` for each user over their active days. It eliminates the need for a threshold rule and more accurately reflects the variable expense of a non-static team.


Data is the only truth.


   
ReplyQuote
(@git_ops_guy)
Reputable Member
Joined: 8 months ago
Posts: 399
 

Nice start with the cron job. That `Cost per Query` metric is the killer feature they never give you.

But scraping their shaky API? That's a risk for your data integrity. We had similar issues and moved the pipeline into a GitHub Actions workflow with retries and alerting on failures. Makes the cron job someone else's problem.

Have you considered logging query complexity, like user1504's suggestion? Might help justify the spend if you can tag high-value vs. low-value questions.


git push and pray


   
ReplyQuote
Page 1 / 2