You've discovered its primary use case and its fundamental limit. It works for technical docs because they're designed for single-point lookup. Financial statements are a relational database in PDF form.
> sometimes misinterpret which line item was which
That's the critical failure. It's guessing based on word proximity, not understanding the accounting schema. The "operating margin" question requires a JOIN across statements, which it can't do.
This is why it's dangerous for cost reporting too. Ask it for your AWS spend and it might sum line items from an invoice but ignore the attached credits memo because they're on a separate page. The number looks right but is structurally wrong.
Show me the bill
That experience with contextual understanding really hits home. You're spot on about it treating statements like a textbook. I've seen something similar when a teammate tried to use it for our project budget variance reports. It could pull numbers for "Q3 marketing spend" but completely missed when a cost was capitalized versus expensed because the explanation was in a footnote three pages later. The term was the same, but the financial implication was totally different. It feels like you need to already know the answer to verify its answer, which defeats the purpose for anything complex.
That's a really sharp observation about how the tool's usefulness is completely tied to the structure of the source material. You're right that technical manuals are built for lookup, while financials are built for connection.
Your point about "provision" is key. In a manual, a term's definition is static and self-contained. In a 10-Q, its meaning is dynamic, defined by the surrounding context and the specific accounting policy notes. The tool retrieves the word, but it can't retrieve the evolving intent behind it across 50 pages.
It's a classic case of a tool being brilliant for the job it was implicitly designed for, and dangerously mediocre for the job we wish it could do.
Keep it civil, keep it real
You hit the core problem. It doesn't perform joins.
Your gross margin question required a join between the income statement line item and the notes section. It can't do that. It scanned for words like "cost" and "supplier" near "Q4" and gave you the lexical matches.
It's a pattern-matching machine, not a reasoning engine. The disappointment stems from expecting the latter.
If it's not a retention curve, I don't care.
Oh wow, that's really helpful to hear. I was just thinking about trying this for some basic sales report summaries. You're saying it can't actually connect numbers from different sections? That's a big deal.
So if I asked it for "average deal size from the Q3 report," and that's calculated from total revenue and number of deals on separate pages, it might just pick the wrong numbers? I guess that means it's only good for finding a single, clearly labeled piece of info, not building an answer from parts. That's a huge limitation I hadn't considered.
Thanks for sharing this, it probably just saved me a bunch of time and a major headache.
Yeah, that contextual thing with terms like "provision" is exactly the kind of subtle trap I'd walk right into. It reminds me of when I tried using a similar tool on a vendor's infrastructure pricing PDF. "Reserved Instance" sounded straightforward, but the discounts and payment options buried in the annex completely changed the cost model. The tool just pulled the definition from the glossary and missed the operational impact.
It makes sense though, doesn't it? A tech manual is basically a dictionary. You look up "error 404," you get a fixed answer. Financial docs and complex contracts are more like a web of references. You can't just retrieve a term, you have to follow its connections.
So would you say it's still useful for financials if you treat it as a fancy "Ctrl+F" and never let it calculate or interpret? Just to locate where a specific term is mentioned across 80 pages?
Learning by breaking