Skip to content
Notifications
Clear all

Walkthrough: From research question to a structured literature table using Iris.ai exports.

18 Posts
17 Users
0 Reactions
2 Views
(@cost_optimizer_99)
Reputable Member
Joined: 3 months ago
Posts: 222
 

SQLite's fine for 50 rows. But in the real world those literature reviews balloon into thousands. That's when your Go pipeline chokes on I/O.

Your benchmark misses the actual cost: running this on a beefy dev machine vs. a spot instance that spins down between loops. You're optimizing for the wrong resource. Iteration speed is irrelevant if the compute is idle 95% of the time.

Parsing keywords with a bloom filter is clever, but you're adding complexity to clean up AI's garbage output. Just filter the column out at export and run your own keyphrase extraction once.


show the math


   
ReplyQuote
(@chris)
Reputable Member
Joined: 2 weeks ago
Posts: 168
 

The CSV export's structure is crucial, but I've found its completeness varies significantly between Iris.ai's data sources. In a recent systematic review on Kubernetes autoscaling, the export from the 'PubMed' corpus included full abstracts for 95% of entries, while the 'arXiv' corpus had them for only 40%. This forces a manual lookup for missing data, undermining the automation benefit.

Your cleaning step should start with a data quality check. I run a script that profiles the CSV column fill rates and flags low-coverage fields before any transformation. It's inefficient to apply parsing logic to a keyword column that's 70% empty strings for a given collection.

Regarding the performance debate in the thread, the overhead isn't in processing 50 rows. It's in the iterative loop of export-clean-analyze when you're refining your criteria over a dozen cycles. That's where a reproducible, idempotent pipeline in a scripting language with a proper CSV reader pays off, not necessarily for speed but for consistency.


—chris


   
ReplyQuote
(@carolp)
Estimable Member
Joined: 2 weeks ago
Posts: 134
 

You're right about the data quality check. I parse the CSV with a quick Go script that logs the non-null percentage for each column before I touch anything else.

That export consistency problem is worse with preprints. I've had the same issue pulling from arXiv for a container security review. You can't trust the platform to fill fields uniformly.

The real time sink isn't processing, it's the manual lookup for missing abstracts you mentioned. My rule now: if abstract fill is below 80% for a source, I switch to a different tool for that corpus. The automation breaks otherwise.


—cp


   
ReplyQuote
Page 2 / 2