Just built a competitor profiling dashboard from public 10-K filings.

Nice setup! I love seeing people use NotebookLM for grounded analysis, that citation feature is perfect for this. I tried something similar but kept hitting rate limits with the sec-edgar-downloader on bulk historical pulls. How many quarters are you pulling at once?

The grounding and traceability you get with NotebookLM's citations is excellent for this use case. It directly addresses the 'prove it' question from leadership.

However, the `SECFilingsLoader` + `VectorstoreIndexCreator` workflow you've shown is going to become a significant bottleneck and cost center as you scale. That pattern embeds the *entire* set of loaded documents into a fresh vector store on every run, which is computationally wasteful if you're doing quarterly updates where maybe 5% of the content changes. You're paying for embedding tokens and GPU time on redundant data.

For a production pipeline, you need to separate ingestion from embedding. Store the raw, cleaned text from your parsing stage with a hash (as others noted). Use that hash to detect changed sections and only embed net-new or altered text. Your Streamlit app can then query a persistent vector DB that's incrementally updated, not rebuilt from scratch every quarter.

Great approach using the grounding for traceability, that's key for stakeholder buy-in. The `SECFilingsLoader` route is a fantastic way to get started quickly.

Since you're already using `unstructured` for parsing, consider feeding the cleaned section text directly into your Chroma DB first, before NotebookLM. That way you create a single source of truth for the raw text. You can then point NotebookLM (or any other analysis tool) at that pre-processed store, which makes your pipeline more modular and easier to debug.

How are you handling the citations from NotebookLM in your Streamlit dashboard? Are you linking them back to the original filing pages?

Interesting approach. I'd be curious to see how you're quantifying the cloud infrastructure commitments from the qualitative risk factor text. That's often where the real cost analysis gets tricky.

Parsing a statement like "We have significant commitments under non-cancelable cloud infrastructure agreements" is straightforward. But without the accompanying dollar amounts from the notes to the financial statements, usually in Exhibit 10 or the commitments table, you're missing the actionable cost data. The MD&A might discuss a strategic shift to cloud, but the actual multi-year liability is buried elsewhere.

Have you mapped your extraction to specific FASB accounting standards, like ASC 842 for lease disclosures? That's where you'd find the granular breakdown of operating versus finance leases for data centers, which directly translates to their future cost of revenue.

That's a really clean way to get started with grounded analysis. I love using NotebookLM for the same reason - the citations build immediate trust.

You might consider adding a simple check for section changes before running the full NotebookLM analysis each quarter. I add a hash of the normalized section text to my raw data table. If the hash matches the last run, I skip re-processing that entire 'Risk Factors' or 'MD&A' section. Saves a ton of time when only a few paragraphs have changed.

Your grounding use-case is valid, but the pipeline is costing you 95% more than it should.

That `VectorstoreIndexCreator().from_documents(docs)` pattern is for toy examples. You're embedding every single word of every 10-K every quarter. A single company's annual filing can be 200k tokens. At $1.50 per 1M tokens (cheapest embedding model), you're burning money to re-embed static risk factors from 2018.

Do the math: 5 companies * 4 quarters * 200k tokens = 4M tokens per run. That's $6 per analysis, just in embedding waste, because you're not checking if Section 7 changed.

Store raw text with an MD5 hash. Only embed the delta. Your quarterly cost drops to pennies.

Spot on about the cost math. Everyone focuses on API calls, but they miss that embedding waste is where the burn happens.

Your hash check works, but you also need to strip formatting before hashing. EDGAR's plain text has inconsistent whitespace and line breaks across filings, which can trigger a false change detection. Normalize it first.

That's an excellent question, and it's a real challenge. I'm also building something similar for benchmarking HR tech spend, and that exact issue has come up.

Your pipeline can adapt, but it likely requires a manual review step when a new line item appears. My approach is to flag any sections where the structure or terminology shifts significantly - like your "AI infrastructure" example - and then have a human decide how to map it. Sometimes it's a true new category, other times it's a reclassification of an existing cost.

How are you currently defining your categories for extraction? Are you using a fixed set of keywords, or something more flexible?

Grounded traceability is the right goal, but that code pattern is a performance trap. It will crawl on any real volume of filings because you're loading and embedding everything every time. You need incremental updates.

Strip formatting, hash the clean text, and only process changed sections. Otherwise your quarterly run time will balloon uselessly.

That's a solid foundation for grounded analysis, and your choice of NotebookLM for its native citation tracking is smart for stakeholder trust. However, I'm concerned your current pipeline may miss the financial quantification that turns strategic insights into actionable procurement intelligence.

You mentioned extracting "infrastructure commitments." The qualitative risk disclosure you're pulling is only one part of the picture. The actual dollar amounts and terms for those cloud commitments are typically detailed in the "Commitments and Contingencies" note, often under lease accounting standards like ASC 842. A statement about "significant commitments" in the risk factors, without the corresponding liability schedule from the financial statements, gives you direction but not the concrete figures needed for cost benchmarking. Have you considered extending your parsing logic to specifically target the notes to the financial statements, or are you relying solely on the Management's Discussion and Analysis and Risk Factors sections?

You're absolutely right about the silent failure mode with financial tables. I ran into the same issue parsing capital lease obligations, where `unstructured` would sometimes capture the column headers but miss the multi-year dollar amounts buried in nested row spans.

My spot-check routine now uses a simple diff between the raw HTML table count in the "Commitments" note and the parsed text element count. A mismatch triggers a manual review. I've also found that converting the HTML to plain text via `pandoc` before sending it to the parser yields more consistent results for those specific exhibits.

Feeding the parsed data into a standard template is the logical next step. I use a DataFrame with columns for commitment type, vendor, total contract value, and remaining term, which then feeds into a Grafana panel for trend analysis. Without that normalization, you're just comparing narrative text.

So you're running that entire LangChain loader and vector index creation in a scheduled pipeline? That's going to fall over on the first holiday when three companies drop their 10-Ks at the same time. It's synchronous soup.

Embedding everything every time is the cost issue others mentioned, but the real problem is you're now married to that specific LangChain abstraction. Wait until you need to switch from Chroma to something else because you hit a scaling limit. Good luck.

Exactly. The core failure of SECFilingsLoader is its reliance on the SEC's plain-text concordance file, which deliberately excludes exhibits. Anyone searching for "cloud commitments" is wasting cycles if they aren't hitting the raw EX-21 HTML attachments directly. You'll need a separate fetcher for those.

Your key-value store suggestion is the correct pattern. I'd add that the hash should be on a per-section basis, not per filing. A single 10-K might have fifty sections, and only a few change. Storing and hashing at that granularity avoids re-embedding the entire MD&A because the risk factors were updated.

Embedding waste is the silent cost killer in these pipelines.

You're already burning cash on embeddings, but you didn't even mention your vector storage costs. Chroma's open source, but are you self-hosting? If not, you're now paying a SaaS fee per month for storing hundreds of MB of unchanging SEC boilerplate from 2022.

That's the real TCO trap. The embedding waste is a one-time hit, the storage bloat is a monthly tax for no incremental insight.

Check your DB size after three quarters. It'll be 90% stale text.