As part of our laboratory's initiative to integrate specialized LLM tools into the research workflow, we recently undertook the task of connecting SciSpace (formerly Typeset) to our internal, private reference server. The goal was to allow SciSpace's literature review and querying capabilities to interact with our proprietary repository of papers, datasets, and technical reports, which are not publicly indexed. This process involved several non-trivial steps concerning authentication, API configuration, and data schema alignment. I am documenting my methodology and findings here for others who may attempt a similar integration.
The primary challenge was that SciSpace is designed primarily to interface with public databases like PubMed, Crossref, or its own internal corpus. To integrate a private server, we needed to leverage the "Custom Source" or API-based ingestion features, which are less documented for on-premise deployments. Our private server is a structured document store built on a combination of a PostgreSQL database for metadata and a MinIO instance for PDF storage, with a RESTful API layer on top.
The integration required two main phases: configuring our server to present data in a compatible format, and then instructing SciSpace to query it. Below is a high-level overview of the configuration steps we implemented on our lab server to create a SciSpace-compatible endpoint.
**Phase 1: Lab Server Configuration**
We developed a dedicated API endpoint that mimics the response structure SciSpace expects from its standard sources. The key was to map our internal metadata fields to the schema SciSpace uses (e.g., `title`, `authors`, `abstract`, `publication_year`, `doi`, `url`). We also added a `full_text` field that provides plain-text extracted content from our PDFs on demand.
```python
# Example Flask endpoint snippet for the lab server
@app.route('/api/scispace/search', methods=['GET'])
def search_documents():
query = request.args.get('q')
# Internal logic to search vector DB and SQL database
results = internal_search(query)
formatted_results = []
for doc in results:
formatted_results.append({
"id": doc['internal_id'],
"title": doc['title'],
"authors": doc['authors'],
"abstract": doc.get('abstract', ''),
"year": doc['year'],
"doi": doc.get('doi', None),
"source": "Lab Private Server",
"url": f"{INTERNAL_SERVER_URL}/doc/{doc['internal_id']}",
"full_text_access": f"{INTERNAL_SERVER_URL}/api/text/{doc['internal_id']}"
})
return jsonify({"results": formatted_results, "count": len(formatted_results)})
```
**Phase 2: SciSpace Setup**
Within the SciSpace platform, we used the "Add Custom Source" feature in the organization settings. This required:
* **Endpoint URL:** The URL of our `/api/scispace/search` endpoint.
* **Authentication:** We used a static API key header for simplicity. The key was added in the SciSpace custom source configuration as a request header (`X-API-Key: `).
* **Query Parameter Mapping:** We configured SciSpace to send the user's search query as the `q` parameter.
* **Pagination:** Implemented basic `limit` and `offset` parameters on our server to handle large result sets.
**Key Observations and Pitfalls**
* **Schema Strictness:** SciSpace is relatively strict about the JSON response format. Missing expected fields can cause silent failures. We found that including a `full_text_access` URL that returns raw text was critical for SciSpace's "Explain" and "Summarize" features to work on our documents.
* **Authentication Overhead:** Our initial attempt using OAuth2 was excessive. A simple API key, combined with IP whitelisting of SciSpace's outbound IPs (obtained from their support), proved sufficient and more maintainable.
* **Performance:** The latency of our internal server directly impacts the user experience in SciSpace. We had to optimize our internal search (introducing a hybrid keyword/vector search using `pgvector`) to keep response times under 2 seconds.
* **Limited Testing Environment:** The custom source feature in SciSpace does not have a "sandbox" mode. Testing required deploying our endpoint to a staging server and making live queries from the platform, which was iterative and somewhat cumbersome.
The integration is now functional and has significantly streamlined the literature review process for our team. However, it remains a custom pipeline that requires maintenance. For organizations considering a similar path, I recommend a thorough assessment of whether the benefits of direct integration outweigh the upkeep cost compared to periodically exporting metadata to a public DOI-based system SciSpace natively supports.