Skip to content
Notifications
Clear all

Just built a competitor profiling dashboard from public 10-K filings.

17 Posts
17 Users
0 Reactions
2 Views
(@amyw)
Estimable Member
Joined: 3 weeks ago
Posts: 191
 

Nice setup! I love seeing people use NotebookLM for grounded analysis, that citation feature is perfect for this. I tried something similar but kept hitting rate limits with the sec-edgar-downloader on bulk historical pulls. How many quarters are you pulling at once?


measure twice, ship once


   
ReplyQuote
(@briank)
Reputable Member
Joined: 3 weeks ago
Posts: 240
 

The grounding and traceability you get with NotebookLM's citations is excellent for this use case. It directly addresses the 'prove it' question from leadership.

However, the `SECFilingsLoader` + `VectorstoreIndexCreator` workflow you've shown is going to become a significant bottleneck and cost center as you scale. That pattern embeds the *entire* set of loaded documents into a fresh vector store on every run, which is computationally wasteful if you're doing quarterly updates where maybe 5% of the content changes. You're paying for embedding tokens and GPU time on redundant data.

For a production pipeline, you need to separate ingestion from embedding. Store the raw, cleaned text from your parsing stage with a hash (as others noted). Use that hash to detect changed sections and only embed net-new or altered text. Your Streamlit app can then query a persistent vector DB that's incrementally updated, not rebuilt from scratch every quarter.


p-value < 0.05 or bust


   
ReplyQuote
Page 2 / 2