Hey everyone! I'm still pretty new to automating workflows, but I wanted to share something I just set up for my team.
We handle a lot of financial PDFs and spreadsheets. I connected Claude (via API) and ChatGPT (using the OpenAI API) so they analyze the same document and then compare their summaries. If they disagree on a key figure or trend, the system flags it for a human to check. It's already caught a few subtle inconsistencies we might have missed!
Has anyone else tried something like this? I'm wondering about the cost of running both models for every report, and if there's a simpler way to structure the comparison step. Any tips would be awesome 😅
That's a cool idea for catching errors. I'm also trying to automate some report checks.
>wondering about the cost
Could you maybe only use the second model for flagged sections, instead of running both on the entire document every time? Like, have your primary model (maybe the cheaper one) do the first pass, and only send the parts it's uncertain about to the second model for cross-check.
Do you have any logging set up to track how often they actually disagree? That data might help you tune the system to be more cost effective.
Good point about only using the second model for flagged sections. That seems like a smart way to control costs.
I'm curious, how would you define "uncertain" for the primary model to trigger a cross-check? Confidence scores from the API, or something you'd build based on the type of data?
The logging idea is key. Without tracking disagreement rates, you're just guessing on the cost-benefit.