Just wrapped up a systematic review using Iris.ai. Input: 500 PDFs from PubMed/arXiv on "adversarial ML defenses." Goal: map the methodologies, extract key claims, and identify consensus gaps.
**Timeline**
* **Setup & Training (Week 1):** Defining concepts ("adversarial robustness," "gradient masking") and training the AI Smithery extractor took ~3 days. The learning curve is real.
* **Full Processing (Week 2):** Ran the full corpus. Took about 36 hours. 12 papers failed parsing (bad formatting).
* **Validation & Export (Week 3):** Spent most of the time manually checking extracted data against source PDFs. Export to CSV was straightforward.
**Key Pain Points**
* **PDF Parsing is Brittle:** Older or column-formatted PDFs (common in older conferences) often yield garbled text. Manual correction required.
* **Concept Drift:** The tool's understanding of your defined concepts isn't perfect. You get false positives on "adversarial example" if the paper just mentions it in related work. Requires constant tuning of your "context filters."
* **No Network Graph Output:** You can extract entities and relationships, but building a visual citation or concept network isn't native. Had to export and use Gephi.
* **Cost Creep:** The "AI Smithery" add-on for custom extraction is essential for serious work, but it's a separate cost on top of the base subscription. Metered usage can spike if you're iterating on training.
**Bottom Line:** Useful for accelerating the initial sift and data extraction phase of a large literature review, but it's not a "set and forget" system. Budget significant time for setup, validation, and manual cleanup. For a team doing this regularly, it might justify the cost. For a one-off project, the overhead is considerable.
-dk
Trust but verify, then don't trust.