I've built a bot that listens to sales calls via a Zoom webhook, transcribes them with Whisper, and then summarizes the key points, objections, and next steps into our Salesforce. The workflow itself is solid—it works exactly as designed.
The problem is cost structure. I'm hitting Poe's message caps hard. A single sales call can be 30+ minutes of audio. My chain is:
1. Transcribe audio (1 message)
2. Chunk transcript (1 message)
3. Summarize each chunk (2-3 messages)
4. Synthesize final summary and extract structured data (1-2 messages)
That's 5-8 "messages" for *one* call. At ~20 calls a day, I'm blowing through the $20/month plan in a week. Scaling up is not cost-effective.
I need alternatives that decouple cost from per-message/per-API-call pricing. My non-negotiables:
* Must output structured JSON to push to CRM/ERP.
* Must handle audio transcription or integrate with a dedicated service (like AssemblyAI).
* Prefer an API-driven, middleware-friendly approach.
I'm evaluating:
* **Self-hosted LLM (via Ollama, etc.)**: Upfront dev cost, but then only infra costs. Worried about summary quality vs. GPT-4.
* **Direct OpenAI API + a middleware platform (like Workato/Celigo)**: Bypass Poe altogether. Pay per token, which might be more predictable.
* **Specialized sales intelligence APIs (like Gong's API)**: Expensive, but purpose-built.
Has anyone else hit this scaling wall with Poe for data-intensive bots? What's your stack for processing high-volume, long-form content into structured systems without getting killed on per-"message" fees?
Code-wise, my current bot logic for the summary step is straightforward:
```javascript
// This is the chunk summarization prompt - called multiple times per call
const summaryPrompt = {
"system": "Extract key points, customer objections, and agreed action items.",
"user": `Transcript chunk: {{transcript_chunk}}
Output a JSON object with: summary, objections[], next_steps[]`
}
```
The per-call token count is high, but the per-"message" cost on Poe is the killer.
Integration is not a project, it's a lifestyle.
Self-hosted LLM is the obvious move for cost decoupling, but that quality drop from GPT-4 is real for nuanced tasks like extracting objections. Have you considered a hybrid approach?
Use a cheap, fast local model (maybe Llama 3.1 8B via Ollama) for the initial chunk summarization. Then, send *only* those condensed chunks to GPT-4 for the final synthesis and JSON extraction. You'd cut your expensive API calls down to 1-2 per call instead of 5-8.
The transcription itself is a separate cost sink. Offloading that to a dedicated service like AssemblyAI might still be worth it, but you could also run Whisper.cpp locally if you've got the GPU.
System calls per second matter.
The hybrid approach is smart on paper, but you're adding a major engineering cost and a new point of failure. You now need to manage two model pipelines, their respective error handling, and the handoff logic.
My bigger issue is the data quality assumption. You're betting a cheap local model's summarization won't distort or omit a critical nuance that the final GPT-4 synthesis needs. For sales objections, that's a risky bet. Garbage in, garbage out still applies, even if the final step is expensive.
Have you actually tested the quality degradation on a set of held-out calls with this method? I wouldn't roll it out without that benchmark.
Garbage in, garbage out.