Just saw the announcement about ResearchRabbit adding direct arXiv and bioRxiv integration. This is a game-changer for real-time literature discovery, essentially turning the app into a live stream of preprint publications.
From a data pipeline perspective, I'm incredibly curious about how they're implementing this. Are they:
* Polling the arXiv API on a schedule, or have they built a proper event-driven ingestion layer?
* Handling the schema differences and metadata formats between arXiv and bioRxiv?
* De-duplicating entries when preprints later get published in journals already in their system?
The trade-offs here are fascinating. A simple scheduled pull is easier to build, but you introduce latency—missing that crucial window when a hot new preprint drops. A streaming approach is more complex but aligns perfectly with the "rabbit hole" discovery metaphor they use.
Has anyone kicked the tires on this yet? I'm wondering about the freshness of the data feed and how it's integrated into the existing recommendation graphs. Does it feel like a real-time update, or more like a daily batch job?