I've been prototyping a workflow in AgentGPT where the agent ingests and summarizes batches of technical PDFs (research papers, mostly). The pipeline works, but I'm hitting a consistent problem: memory usage spikes dramatically during the PDF parsing and text extraction phase, sometimes causing the entire process to stall.
My setup uses a common pattern: fetch PDF → parse with PyPDF2 (tried pdfplumber too) → chunk text → send to LLM for summarization. The memory seems to balloon during the parsing stage, even with modest 10-page documents. I suspect it's holding the raw byte data, extracted images, and text all at once before cleanup.
Has anyone else tackled this? I'm thinking about the trade-offs:
* **Streaming the PDF bytes:** Instead of loading the whole file, could we process it page-by-page in a true streaming fashion? Is there a library that supports this well?
* **Aggressive cleanup:** Explicitly `del` the raw bytes and parser object after text extraction, maybe even forcing garbage collection.
* **External processing:** Offloading PDF parsing to a separate, memory-limited microservice that just returns the text.
* **Different libraries:** I've heard `pymupdf` (fitz) can be more efficient? Not sure about its memory profile.
I'm especially curious if anyone has integrated a solution that keeps the overall AgentGPT agent lightweight, treating the PDF parsing as a stateless, memory-efficient transformation step. What worked for you?
—Claire