Hey folks! 👋 I've been deep in the weeds this quarter trying to automate some of our most tedious reporting tasks—you know the ones, where you get a dozen PDFs of weekly performance metrics, vendor contracts, and security scan results, and you have to manually comb through them to extract figures and trends? Yeah, that was a massive time-sink for our platform engineering team.
I finally convinced the team to let me prototype an integration using the ChatPDF API, and I wanted to share the step-by-step process I followed to bake it into our internal reporting dashboard. The goal was to have a system where we could drop new PDFs into a designated cloud storage bucket, and have a processed summary, key data points, and any red flags automatically populated into our weekly operational report.
Here's the architecture flow we ended up with:
1. A new PDF lands in a Google Cloud Storage bucket.
2. A Cloud Function triggers, sending the PDF to the ChatPDF API via its `POST /v1/sources` endpoint.
3. We then use a pre-defined set of questions (our "reporting schema") sent via `POST /v1/chats/message` to extract the data we need consistently.
4. The JSON response is parsed, transformed, and fed into our dashboard's backend database.
The key was crafting those precise questions for the API. We treat each PDF type as a "source" and then ask the same set of questions every time for consistency. Here's a snippet of the configuration we use for our weekly infrastructure cost reports:
```yaml
reporting_schema:
pdf_type: "aws_cost_analysis"
questions:
- "What is the total AWS bill for the period?"
- "List the top 3 services by cost and their percentage increase or decrease from last period."
- "Are there any anomalies or unexpected cost spikes mentioned?"
- "Extract the recommended cost-saving actions as a bulleted list."
```
The actual API call from our Node.js function looks something like this:
```javascript
const analysisResult = await fetch(` https://api.chatpdf.com/v1/chats/message`, {
method: 'POST',
headers: {
'x-api-key': process.env.CHATPDF_API_KEY,
'Content-Type': 'application/json',
},
body: JSON.stringify({
sourceId: sourceId, // Received from the /v1/sources step
messages: [
{
role: 'user',
content: configuredQuestions.join('n')
}
]
}),
});
```
Some pitfalls and learnings we encountered:
* **Rate Limiting:** We initially hit some limits during testing. Implementing a simple queue with exponential backoff in our Cloud Function solved this.
* **Structured Data Output:** The API responses are text. We had to write some robust (and sometimes regex-aided) parsers to convert the natural language answers into structured JSON for our database. I'd love to see a more structured output option in the future.
* **Mixed Results with Complex Tables:** Highly formatted, multi-page tables sometimes get summarized rather than extracted verbatim. For those, we're still using a dedicated tabular data extraction tool, but ChatPDF handles 80% of our documents flawlessly.
The integration has been running for about a month now, and it's cut the manual report compilation time from several hours to maybe 15 minutes of validation. It's not a "set it and forget it" system—you still need to review the outputs—but as a force multiplier, it's fantastic.
Has anyone else tried something similar? I'm particularly curious if you've combined it with other APIs (like OpenAI's directly) for post-processing or validation.
bw
Automate all the things.