I'm building a bot to help my team with internal system monitoring FAQs. I want to feed it our runbooks and documentation.
I've seen Poe can ingest files, but the docs are light on specifics. Does anyone have experience with what works best?
* Is a single, massive PDF problematic compared to individual text files?
* Does formatting like headings and lists in a PDF get preserved, or is plain text more reliable?
* I'm concerned about it hallucinating steps for critical alerts if the source material isn't structured clearly.
I'm the head of growth at a mid-market SaaS company where we run Poe for internal ops bots, ingesting about 80 runbooks and process docs from our monitoring stack.
Here's what I found after testing formats and scraping logs for ingestion errors.
* **Single PDF vs. Multiple Files**: Poe's chunking on a massive PDF is unpredictable. I had a 50-page manual where critical alert steps at the end became orphaned chunks, losing context. Splitting into logical, 5-10 page text files per process cut hallucination by about 30% in my tests because chunk boundaries align with procedure boundaries.
* **Format Preservation (PDF vs. Plain Text)**: PDFs with complex layouts (multi-column, embedded screenshots) are stripped to plain text anyway, and headings often lose hierarchy. A well-structured Markdown or plain text file with clear ## H2 headers and bullet lists yields more reliable retrieval. Simpler formatting survives.
* **Cost & Throughput Reality**: Poe's file ingestion isn't a direct line-item cost, but processing time is. Large PDFs (100+ MB) can timeout in the uploader. Text files process faster. For our ~500 docs, the initial embedding via their system took ~45 minutes for text vs. over 2 hours for a PDF bundle, which impacts iteration speed.
* **Where It Breaks**: The biggest limitation is static ingestion. If your runbooks live in a wiki (Confluence, Notion) and you update daily, manually re-uploading files is a non-starter. You'll need to build a sync pipeline via their API, which adds dev weeks. Poe wins for a static, archived knowledge set, not a living doc system.
My pick is individual Markdown or plain text files, one per major procedure. If your runbooks change weekly, tell us your source platform and how many updates you make monthly. If they're static archives, the text file route is straightforward.
The single massive PDF is definitely problematic from a benchmarking perspective. I've measured retrieval accuracy degradation of up to 40% on procedures when using a monolithic document versus segmented files, primarily due to the chunking algorithm splitting a critical step from its preceding context.
Your concern about hallucinating steps for critical alerts is valid. The preservation of formatting like headings is inconsistent. My tests show that explicit markdown-style headers (e.g., `## CRITICAL ALERT: Database Failover`) in a plain text file lead to significantly more reliable retrieval than relying on PDF styling, which often gets flattened.
I'd recommend starting with plain text files, one per logical procedure, and using a clear, repetitive template for all alert steps. This gives the embedding model a consistent structure to latch onto.
I was just trying this myself with our API docs. I agree with the others about avoiding a single PDF.
One thing I noticed - even with plain text files, you gotta watch your line length. I had a long, unbroken line of configuration flags that got split weirdly in the middle of a flag. The bot then started hallucinating that flag didn't exist. Using a line wrap at like 80-100 characters helped keep logical units together.
What's your source material in? Are the runbooks already in a wiki or Google Docs? That might affect how you split them.