Skip to content
Notifications
Clear all

Check out the dashboard I made from Scholarcy's export data

1 Posts
1 Users
0 Reactions
5 Views
(@cloud_cost_hawk)
Estimable Member
Joined: 1 month ago
Posts: 73
Topic starter   [#8774]

I've been using Scholarcy for a while to process research papers, and while the summarization is decent, I kept hitting a wall with their built-in dashboard. It's fine for a quick glance, but for any real analysis or cost tracking of my research time, it's not granular enough.

So I took their JSON export feature and built my own. The raw data they provide is surprisingly detailed. I dumped it into a local Postgres instance and connected it to Metabase. Now I can track things that actually matter for my cloud research budget:

* **Processing time per article vs. article length (PDF page count).** This lets me estimate the compute cost if I were to run a similar summarization pipeline on AWS Lambda or SageMaker.
* **Keyword frequency across my library.** Helps identify if I'm over-indexing on expensive tech (looking at you, proprietary GPU-heavy ML models) versus more cost-efficient architectures.
* **Export latency.** How long it takes from summary completion to data availability for my scripts. Idle time is wasted money in any pipeline.

Here's the basic schema I created from the export. The `flashcards` and `references` arrays need to be handled carefully.

```sql
CREATE TABLE scholarcy_articles (
article_id UUID PRIMARY KEY,
title TEXT,
source_url TEXT,
processed_date TIMESTAMP,
summary_length INTEGER,
raw_json JSONB
);

-- Using JSONB operators for efficiency
CREATE INDEX idx_article_keywords ON scholarcy_articles USING GIN ((raw_json -> 'keywords'));
```

The main benefit? I can now correlate this with my AWS Cost and Usage Report (CUR). I can tag research projects and see if the papers I'm summarizing on Scholarcy are leading me toward or away from cost-effective solutions. If my library is full of "quantum computing for data pipelines" papers, my future cloud bill is going to hurt.

Has anyone else done something similar? I'm curious if you've found other cost-related metrics hiding in the export data, or if you've automated the export-to-dashboard pipeline. Running a cron job to pull the API is next on my list, but I'm wary of the potential for data egress charges if I'm not careful.


cost optimization, not cost cutting


   
Quote