Hey everyone! I've been diving into the SciSpace logs from my team's usage over the past quarter. We're a research group using it heavily for literature reviews and annotation, and I kept hearing different people swear by totally different workflows. I got curious about what patterns were actually emerging.
So, I pulled the usage logs via their API (which was pretty straightforward, thankfully) and built a simple dashboard in Metabase. I wanted to track things like: most-used annotation colors (are we using red just for critical flaws?), average time between opening a paper and making the first highlight, and even which paper sections (intro, methods, results) we tend to annotate the most.
The initial findings are kind of fascinating! For example, our team uses the yellow highlighter for "to cite" way more than anything else, and we barely touch the green. Also, most of our annotations happen in the first 15 minutes of reading a paper. It's making us rethink our internal guidelines.
My question for you all: has anyone else done something similar? I'm especially wondering about the data pipeline side. I did a simple daily Airflow DAG to pull the logs and load them into a small Postgres table, but I'm sure it's not the most robust setup. How would you handle incremental updates or if the API schema changes? Also, any ideas for other cool metrics to track from this kind of usage data? I feel like I'm just scratching the surface.
-- rookie
rookie
Interesting project. I've done similar instrumentation for a few internal tools, mostly around code review latency rather than annotation behavior, but the pipeline architecture overlaps.
> simple daily Airflow DAG to pull the logs and load them into a small Po
One thing that bit me early on with a similar approach: SciSpace's API response times can degrade under heavy usage, and a single daily batch pull might miss ephemeral annotation events if the API paginates and you get a timeout mid-collection. I ended up adding a checkpointing mechanism with a small SQLite cache to resume failed pulls, plus a retry with exponential backoff. Not sure if that's relevant for your volume, but worth checking whether your DAG is idempotent.
Also curious what you're using for the destination - you mentioned Metabase, but what's the underlying store? If it's Postgres, the batch insert pattern from Airflow can be tuned with a bit of COPY rather than row-by-row inserts.
benchmark or bust
Metabase and a simple DAG is fine for a pet project. But "making us rethink our internal guidelines" based on a daily batch pull is premature. You're seeing *what* but not *why*. Logs miss intent.
Check your data freshness. That "first 15 minutes" metric is skewed if your daily job fails and you're analyzing stale events. You didn't even finish your sentence about the Po.
`> simple daily Airflow DAG`
Simple is fragile. Hope you're not just `INSERT`-ing. Idempotence isn't optional, even for "small" data. Are you handling late-arriving data from the API, or just assuming yesterday's logs are complete? Seen it blow up every time.
SQL is enough
"Simple daily Airflow DAG" is what you start with before it breaks. Everyone's pipeline is simple until their API throttles them or returns duplicate events.
You mentioned loading into a small Po... I'm assuming a small PostgreSQL instance? If you're just doing raw inserts, you'll have duplicate data after the first retry. Make your loads idempotent from day one. Use a staging table and a merge, or at least track the last successfully ingested event ID.
Your yellow highlighter finding is interesting, but how stale is that data? If your DAG fails silently for two days, your "first 15 minutes" metric is garbage. Add task failure alerts before you trust any findings.
garbage in, garbage out
Interesting project, and I think the behavioral insights you're uncovering are more valuable than the technical pipeline discussions that followed. That said, the pipeline is the foundation for trustworthy insights.
Your "first 15 minutes" finding is a perfect example of a metric that needs a reliable data stream. If your daily DAG fails silently, you're not just missing data, you're introducing a systematic bias into your analysis. The events from a failed day will arrive later, potentially being attributed to the wrong date and distorting your time-to-first-annotation calculation. Implementing a simple watermark table to track the last successfully ingested event timestamp, paired with failure alerts, is non-negotiable before you base any process changes on this.
On the annotation patterns themselves, have you correlated color usage with document section or publication stage? We found that "red for critical flaws" was true in later-stage review, but during initial screening, red was often used for methodological keywords. The intent, which logs don't capture, differed completely. Your yellow-for-citing pattern could be a powerful prompt for a shared Zotero or citation plugin workflow, turning an observed behavior into a team enablement tool.
You're still assuming the data itself is worth this much effort. Garbage in, gospel out. A "simple watermark table" just formalizes the assumption that the API logs are the source of truth for behavior. They're not. They're a proxy, and a flaky one.
Even if you fix the pipeline, you're measuring clicks, not decisions. You'll end up optimizing for dashboard metrics, not better research. I've seen teams waste months building reliable pipelines for utterly meaningless signals.
And now you want to correlate colors with sections? That's just more shiny distraction.
If it ain't broke, don't 'upgrade' it.
I agree that data quality is a prerequisite, but dismissing the whole exercise as measuring "clicks, not decisions" is a false dichotomy. The goal isn't to replace qualitative feedback, it's to find discrepancies between assumed and actual workflows.
Your point about optimizing for dashboard metrics is valid, though. I've seen teams create elaborate color-coding rules just to make a chart look consistent, which defeats the original purpose. The data should inform a conversation, not become the conversation.