SciSpace's annotation export is broken for LaTeX users. Any workarounds?

The API route is your only realistic path forward if you want to maintain the hierarchy. I've had to do this for a similar platform. The CSV is a lossy export, and once the structure is flattened, you can't reliably reconstruct it without a separate metadata call.

Their API likely serves JSON, which will preserve the Unicode characters as they were stored. The problem will be the `cite{}` references - the API might escape those as HTML entities. Write a small script in something like Go or Python that fetches via the API, then runs a series of regex replacements to convert `>` and `<` back to braces, and handles the Greek letter encodings. It's a dirty fix, but it's a one-time cost versus manual reformatting.

Pandoc will struggle because it needs semantic structure, which the CSV lacks. You'll spend more time massaging the CSV into a valid intermediate format than you would just hitting the API directly, even with sparse documentation.

Agreed on the API being the primary path, but there's a cost factor often missed. For a team-based workflow, that one-time script becomes a maintenance liability. Every time SciSpace updates their API fields or authentication method, your export pipeline breaks and requires developer time to fix. That's a hidden TCO.

Your point about escaped characters is correct. However, using regex for HTML entity replacement can be fragile if the API's output format changes. A more stable method is to use a proper HTML or XML parser library in your scripting language of choice. This handles nested or unexpected entity encodings that regex might miss.

For smaller projects, the manual reformatting from CSV might actually have a lower total time investment than building and maintaining an API integration. It depends entirely on the volume of annotations and the expected lifespan of the manuscript project.

The API route is the only scalable option for preserving hierarchy, but I've found their API documentation often lags the actual implementation. Before committing to a script, you should directly inspect the raw API response for a sample annotation. Use a browser's developer tools while logged into SciSpace, monitor the network tab, and look for XHR calls to endpoints containing "annotation" or "note". This will show you the exact JSON structure and character encoding before you write a single line of code.

If the API escapes LaTeX special characters, a simple HTML parser like Python's `html.parser` module can decode entities back to `{` and `}` far more reliably than regex. For the Unicode Greek letters, you'll need a targeted translation dictionary, mapping the specific Unicode code points you're seeing back to `alpha`, `beta`, etc.

The real cost, as another user noted, is maintenance. Your script becomes a critical but fragile data pipeline. Budget for it breaking at least once per major platform update. For a team, that ongoing overhead might indeed justify a fallback to Zotero for the export-specific part of your workflow, even if it feels like a step backwards.

Their annotation system is fundamentally not built for programmatic use beyond their own walled garden. The API is your only real option, but as others have hinted, it's a trap.

You said their API docs are sparse. That's a huge red flag. A vendor with sparse docs for core data means they haven't made a commitment to that data structure as a stable product. Your "one-time" script becomes a permanent maintenance liability, breaking with every platform update.

Abandoning their system for Zotero doesn't defeat the purpose - the purpose is a reliable workflow. Their tool is failing at a critical juncture. The total cost of hacking around this now will exceed migrating your notes to a tool with proper BibTeX/LaTeX support from the start.

You're right about the CSV being a dead end for hierarchy. Pandoc can't invent structure that isn't there.

> using their API directly

That's your only shot. Before you write any script, confirm the data structure. Use your browser's network tab while loading your library page. Find the API call that returns your annotations and check the actual JSON.

If the fields are there, a 50-line Python script with `requests` and `html.parser` for entity decoding can work. Build a dict for Greek letter mapping.

But user1562 has a point: if the docs are that sparse, you're signing up for breakage. Evaluate if the time to build and maintain that script exceeds the pain of migrating to a tool with native BibTeX export now.

You've perfectly summarized the technical steps and the business risk. The "50-line Python script" estimate is a bit optimistic, though, in my experience. Setting up the authentication flow alone (OAuth, tokens, session handling) can blow past that if you need it to run unattended.

My addition is on the maintenance point. If you do go the script route, don't write it just for yourself. Structure it as a clear, documented module where the core translation logic is separate from the API client calls. That way, when the API endpoint changes, you only have to fix one small part instead of untangling a mess of string operations and network code. It turns a breaking change from a crisis into a 30-minute update.

Totally agree on the "50-line script" optimism. For a one-off, yeah, maybe. But for a team? You're immediately looking at a config file for credentials, error handling for rate limits, and a logging setup.

> Structure it as a clear, documented module where the core translation logic is separate from the API client calls.

This is the key. I'd take it a step further and mock the API client during development. That lets you test your translation layer (Greek chars, entity decoding) without hitting their servers at all. Then, if the API shape changes, you update the mock to match the new response and fix your logic against it. It's a bit more upfront work but saves your sanity later.

The OAuth point is real. I've seen people sidestep it by just copying a session cookie from the browser into a script config for personal use, but that's brittle and a security no-no for a shared tool.

Absolutely, the mock-first approach is such a great habit for longevity. It's saved me more times than I can count when working with poorly documented APIs. You can catch breaking changes in your CI pipeline *before* your whole team's workflow grinds to a halt.

The session cookie workaround is a real dilemma, isn't it? For a solo user in a pinch, it gets the job done, but the moment you involve a team, you're trading a known secure protocol (OAuth) for a brittle hack that constantly expires. That's a quick path to shadow IT and wasted hours.

Mocking the API client is indeed crucial, but I'd add that you should also version-control your mock data alongside the script. When the API breaks, you can diff the new, actual response against your saved mock to pinpoint the exact field or structure change. This turns debugging from a black box into a targeted update.

The session cookie trade-off is steep. Beyond security, it creates a single point of failure: the script only runs as long as your browser session is valid, which for some platforms could be a matter of hours. For a team, that means someone is inevitably on cookie-refresh duty, which completely defeats the automation goal.

> version-control your mock data alongside the script

That's a disciplined approach, but you need a process for updating the mock. If you're diffing a new API response, you have to capture it first, which means the live script has already broken. You need a separate, manual process to fetch a fresh sample before your automated job runs again, or the pipeline is down until you intervene.

The cookie expiry problem you mentioned is the same for any manual authentication step, including fetching that new sample for the mock. It creates operational drag that scales poorly. A better mitigation is to schedule a monthly manual run to refresh the mock data as part of your maintenance checklist, treating it like a dependency update, rather than waiting for a failure.

Monthly mock refresh is smart. The key is making that process itself scripted. My team's rule: the mock refresh script is the only part allowed to use the session cookie. It dumps a sample, you review the diff, commit it, and the main automated job uses the mock with a stable, token-based auth that's already broken. That way the fragile manual step is confined to one place.

The CSV is structurally incapable of preserving the relationships you need. Pandoc is a format translator, not a data model reconstructor - it can't rebuild hierarchy from a flattened table.

Your real choice is between the API and a new tool. The API's sparse documentation is a direct signal about their engineering priorities; annotations are a display feature, not a first-class data product. A script will be a permanent maintenance fixture.

Given the volume you mention, the time invested in building a robust parser and mock framework will likely exceed migrating to Zotero now. The unified system promise breaks if the export workflow isn't part of the unity.

If you're staring down hundreds of notes, the rebuild time for a proper script might already match the migration time to something like Zotero. I had a similar issue with another tool.

> their API docs are... sparse

Did you check if there's any browser extension or existing Greasemonkey script that scrapes the annotations from the page HTML? Sometimes the data is more accessible there than in the official export.

Yeah, the browser extension idea is clever - I hadn't thought of that. Could save a ton of time if it works.

But scraping the page might break too if they change the frontend. Is that usually more stable than an official API? Seems risky for long term use.