My script to batch-process PDFs with Claude and log all responses

Another week, another team at work proudly announced they were building a "RAG pipeline" with a vector database, a dedicated inference endpoint, and a fancy orchestration layer to... answer questions about their internal PDF manuals. The budget discussion was north of $20k a month. I nearly spat out my coffee.

Listen, half the time you don't need a sledgehammer to crack a nut. If your core need is simply to process a batch of PDFs through an LLM like Claude and systematically capture the outputs, you can do it with about 150 lines of Python and the API. No Kubernetes, no message queues, no "embeddings microservice." Just a script that works.

I built this because I got tired of one-offing it in the console. It's stupidly simple: point it at a folder of PDFs, ask your question or set your instruction, and it chews through them, logging every response with metadata. The primary value is the *audit trail*—you can always see which PDF generated which answer.

Here's the core of it. It uses `pypdf` to extract text (keep it simple) and the Anthropic API. The logging is just JSON lines; easy to import into anything later.

```python
import os
import json
import time
from pathlib import Path
from pypdf import PdfReader
import anthropic

class PDFBatchProcessor:
def __init__(self, api_key, model="claude-3-haiku-20240307"):
self.client = anthropic.Anthropic(api_key=api_key)
self.model = model
self.logs = []

def extract_text(self, pdf_path):
"""Naive text extraction. Good enough for most docs."""
reader = PdfReader(pdf_path)
text = ""
for page in reader.pages:
text += page.extract_text() + "n"
return text.strip()

def process_pdf(self, pdf_path, system_prompt, user_prompt_template):
"""Sends extracted text to Claude, returns structured response."""
filename = Path(pdf_path).name
print(f"Processing: {filename}")

raw_text = self.extract_text(pdf_path)
# Insert the PDF text into the user prompt template
user_prompt = user_prompt_template.format(pdf_text=raw_text)

response = self.client.messages.create(
model=self.model,
max_tokens=4000,
system=system_prompt,
messages=[{"role": "user", "content": user_prompt}]
)

log_entry = {
"timestamp": time.time(),
"file": filename,
"input_prompt": user_prompt_template,
"system_prompt": system_prompt,
"response": response.content[0].text,
"model": self.model,
"usage": {
"input_tokens": response.usage.input_tokens,
"output_tokens": response.usage.output_tokens
}
}
self.logs.append(log_entry)
return log_entry

def batch_process(self, pdf_directory, system_prompt, user_prompt_template):
"""Main loop. Iterates over all PDFs in directory."""
pdf_dir = Path(pdf_directory)
for pdf_file in pdf_dir.glob("*.pdf"):
try:
self.process_pdf(pdf_file, system_prompt, user_prompt_template)
# Be a good citizen with rate limits
time.sleep(1)
except Exception as e:
print(f"Failed on {pdf_file.name}: {e}")

def save_logs(self, output_file="claude_pdf_logs.jsonl"):
"""Persists all logs as JSON Lines."""
with open(output_file, 'w') as f:
for log in self.logs:
f.write(json.dumps(log) + 'n')
print(f"Saved {len(self.logs)} logs to {output_file}")

# Example usage
if __name__ == "__main__":
processor = PDFBatchProcessor(api_key=os.environ["ANTHROPIC_API_KEY"])
system = "You are a technical analyst extracting key information."
user_template = "Based on the following document, list the top 3 specifications mentioned:nn{pdf_text}"

processor.batch_process("./pdfs_to_process", system, user_template)
processor.save_logs()
```

Key design choices I made:
- **JSONL for logging**: Each response is a separate JSON object on one line. Makes it resilient and easy to append.
- **Haiku as default model**: It's fast and cheap for bulk extraction. Swap in Sonnet or Opus if you need heavier lifting.
- **Naive text extraction**: `pypdf` works for most simple PDFs. If you have scanned docs, you'd need OCR, but that's a separate problem. Don't build for the 5% edge case first.
- **Explicit rate limiting**: A simple sleep between calls. It's not fancy, but it prevents you from hitting rate limits and you're not processing 10,000 PDFs a minute anyway.

The cost for running this on a few hundred PDFs is measured in dollars, not thousands. The real benefit is the clarity: you now have a repeatable process, and you can version your prompts and see how changes affect outputs.

I've used this for:
- Extracting key contract terms from a pile of NDAs.
- Summarizing incident reports from a directory of post-mortems.
- Checking a set of architecture docs for compliance with security principles.

It's not a "pipeline." It's a script. And it often does the job just fine, while the team that insisted on the "production-grade solution" is still stuck in design review for their Kubernetes namespace.