Hey everyone! 👋 I've been deep in a research project lately where I needed to compile data from tables across about two dozen academic papers. Manually copying them was… not an option. I remembered SciSpace (formerly Typeset) has an API, and after some trial and error, I managed to automate the whole extraction process. I thought I'd document my workflow here in case anyone else is trying to pull structured data out of PDFs at scale.
My goal was straightforward: **feed the API a list of PDF URLs or DOIs and get back every table from each paper, neatly formatted and ready for analysis.** Here's the step-by-step approach I landed on.
First, you'll need a SciSpace API key. Once you have that, the core logic involves two main calls for each document:
1. **Submit the document** for parsing.
2. **Poll for completion** and then fetch the results, specifically filtering for table data.
I wrote a Python script to handle the sequence and the polling. The most important part is targeting the `tables` in the response.
```python
import requests
import time
import json
API_KEY = 'your_api_key_here'
HEADERS = {'X-API-Key': API_KEY}
BASE_URL = 'https://api.scispace.io/v2'
def process_document(file_url):
# 1. Submit document
submit_payload = {"url": file_url}
submit_response = requests.post(f'{BASE_URL}/document/url', json=submit_payload, headers=HEADERS)
document_id = submit_response.json().get('id')
if not document_id:
print(f"Failed to submit: {file_url}")
return None
# 2. Poll for processing completion
status = 'processing'
while status == 'processing':
time.sleep(5) # Be polite to the API
status_response = requests.get(f'{BASE_URL}/document/{document_id}/status', headers=HEADERS)
status = status_response.json().get('status')
if status != 'processed':
print(f"Document {document_id} failed with status: {status}")
return None
# 3. Fetch the parsed content, requesting tables
content_response = requests.get(f'{BASE_URL}/document/{document_id}/content?type=tables', headers=HEADERS)
return content_response.json()
# Example usage
pdf_url = "https://example.com/paper.pdf"
tables_data = process_document(pdf_url)
if tables_data:
with open('extracted_tables.json', 'w') as f:
json.dump(tables_data, f, indent=2)
print(f"Extracted {len(tables_data.get('tables', []))} table(s).")
```
A few key learnings from my run:
* **Rate Limiting:** The API has limits. I inserted a 1-second delay between documents in my batch loop to avoid issues.
* **Table Fidelity:** The extraction is good, but complex, multi-header, or merged-cell tables sometimes need manual review. The API returns them as structured JSON, which is fantastic.
* **Content Type Parameter:** The `?type=tables` query in the final fetch is crucial—it filters the massive full document response to just the tables, which is much more efficient.
* **Formats:** You can also upload a file directly if you have it locally, but the URL method worked best for my cloud-based PDFs.
This method saved me literally days of work. I'm now thinking about hooking the output into a small app that compares table structures across papers automatically. If anyone has tried something similar or has questions about the script, I'm all ears! Happy to help.