Hey everyone! I’m new to the data engineering world (mostly been doing basic ETL scripts in Python and SQL) and I’ve been trying to apply some of those structured thinking patterns to my academic research. I just used ResearchRabbit for the first time to tackle a literature review for a small project, and I thought I’d share my week-long experience. Maybe you folks can tell me if I’m on the right track or if there’s a better way to think about this as a pipeline? 😅
My research question was pretty broad: “How are data teams using orchestration tools like Airflow in small-to-midsize companies?” I started by seeding ResearchRabbit with two key papers I already knew. The “similar work” and citation features felt a lot like following dependencies in a DAG—each paper pointed to others, and I could visually map how ideas branched out. It was super cool to see a graph build up over a few days!
I spent the next few days adding papers to collections (like grouping tasks into a pipeline stage), labeling them with custom tags (think: adding metadata), and using the timeline view to see how the field evolved. By day 5, I had this sprawling map. But here’s where I hit a snag: I wanted to export not just the list, but the relationships between papers—like a lineage graph for references. I ended up manually noting the connections, which felt… not scalable. Is there a way to get something more structured out, maybe a JSON or even a network graph file, that I could later load into a database? I’m used to working with defined schemas, so this part felt a bit fuzzy.
Overall, I was really impressed! Going from a vague question to a clear map in a week was awesome. But as someone who likes clean, reproducible workflows, I’m left wondering: how do you all bridge tools like this with more engineering-friendly formats? Do you treat the visual map as a staging area, then build your own layer on top? Would love any tips!
-- rookie
rookie
Ah, the classic "applying data engineering patterns to everything" move. I get the appeal, I really do. Your DAG analogy for citations is charming. But mapping academic literature isn't a pipeline, it's more like untangling a massive distributed monolith with terrible documentation and circular dependencies.
You're treating ResearchRabbit's graph like a neat, directed workflow, but academic influence is messy and non-linear. One paper might be a foundational pillar everyone cites but never actually reads, while another obscure one contains the real insight. The timeline view gives you a false sense of orderly evolution. The reality is more like event sourcing with lots of lost events and competing projections.
I'm curious about that "snag" you hit. Did the map get so sprawling because the tool encouraged breadth over depth? It's easy to chase citations like following service dependencies in a badly documented microservice architecture, ending up with a huge diagram that's impressive but functionally useless. Sometimes you need to draw a bounded context and ignore everything outside it.
🤷