Preparing for a yearly internal review often necessitates extracting a comprehensive, point-in-time snapshot of your Hyperproof data. This is a non-trivial task, as the platform's primary interface is designed for operational use, not bulk data extraction. Based on my analysis of the API documentation and community discussions, there are three primary vectors for export, each with significant trade-offs.
**1. Manual Export via UI (Simplest, Least Flexible)**
This method is suitable for a one-off, high-level summary but is inadequate for a detailed, auditable data dump.
* **Compliance Reports:** You can generate and export PDF/Excel reports for individual controls or frameworks. This fragments data across hundreds of files.
* **Evidence Library:** You can filter and manually select evidence items for export, but this is impractical for thousands of items.
* **Limitations:** No ability to export relational data (e.g., linking controls directly to their evidence, tasks, and comments in a structured table). It's a presentation layer export, not a data layer export.
**2. Hyperproof API (Most Powerful, Requires Engineering)**
This is the only method to obtain a complete, structured dataset. A successful export requires a script that handles pagination, rate limits, and data reassembly. Below is a conceptual Python script outline using the `requests` library.
```python
import requests
import pandas as pd
import time
BASE_URL = "https://api.hyperproof.com"
HEADERS = {"Authorization": "Bearer YOUR_API_KEY", "HP-Organization-ID": "YOUR_ORG_ID"}
def fetch_all_paginated(endpoint):
"""Handles pagination for Hyperproof API endpoints."""
all_items = []
url = f"{BASE_URL}/{endpoint}"
while url:
response = requests.get(url, headers=HEADERS)
data = response.json()
all_items.extend(data.get('items', []))
url = data.get('nextLink') # Follow nextLink for pagination
time.sleep(0.1) # Respect rate limits
return all_items
# Step 1: Fetch core entities
controls = fetch_all_paginated('api/controls')
tasks = fetch_all_paginated('api/tasks')
evidence_items = fetch_all_paginated('api/evidencerecords')
# Step 2: Create DataFrames for analysis
df_controls = pd.json_normalize(controls)
df_tasks = pd.json_normalize(tasks)
df_evidence = pd.json_normalize(evidence_items)
# Step 3: Merge data as needed (example: tasks linked to controls)
# This requires understanding the linking IDs in the schema.
# df_merged = pd.merge(df_tasks, df_controls, left_on='controlId', right_on='id')
# Step 4: Export to a format suitable for review (e.g., Excel with multiple sheets)
with pd.ExcelWriter('hyperproof_annual_review.xlsx') as writer:
df_controls.to_excel(writer, sheet_name='Controls', index=False)
df_tasks.to_excel(writer, sheet_name='Tasks', index=False)
df_evidence.to_excel(writer, sheet_name='Evidence', index=False)
```
**Key API Considerations:**
* You must be an Org Admin to generate an API key.
* The data is returned in JSON, often with nested objects. Normalization is required.
* Critical relationships (Control -> Tasks -> Evidence) must be reconstructed post-export using foreign keys like `controlId`, `evidenceIds`.
* The API does not provide a single "export everything" endpoint. You must orchestrate calls to multiple endpoints.
**3. Hyperproof Data Lake (Enterprise-Only, Most Comprehensive)**
Some enterprise contracts include a data lake feature (AWS S3 or Azure Blob Storage). If available, this is the optimal path.
* **What it is:** Daily snapshots of your entire Hyperproof relational dataset delivered as partitioned Parquet/JSON files.
* **Advantages:** Enables direct querying with Trino/Athena/Synapse. You can run complex historical analyses, track changes over time, and join tables efficiently using SQL.
* **Disadvantage:** Access is typically gated by a premium enterprise agreement.
**Recommendation for a Thorough Yearly Review:**
If the API route is chosen, your export script must sequentially gather and link data from at least the following entities:
1. Organizations, Users, Groups (for context)
2. Control Domains & Frameworks
3. Controls (the core object)
4. Tasks (filter by relevant date range)
5. Evidence Records
6. Comments/Activity Logs (if needed for audit trail)
The resulting dataset should be loaded into a relational database (even a temporary PostgreSQL instance) to allow for the complex joins that a yearly review requires. This approach provides the raw data necessary to answer questions like "What percentage of high-risk controls had overdue tasks at each quarter-end?" which is impossible with UI exports alone.
They always undersell the API, don't they. "Requires Engineering." What it actually requires is figuring out their rate limits and pagination quirks, which the docs treat as an afterthought.
You'll spend more time stitching together a dozen paginated JSON responses than you will writing the actual export logic. And good luck preserving relational integrity if you're hitting separate endpoints for controls, evidence, and tasks. The timestamps are your only real friend for that point-in-time snapshot.
Assume you'll need a simple state store, even just a local SQLite DB, to join it all back together. Otherwise you're just moving the fragmentation from PDFs to JSON files.
Prove it.
That point about relational integrity is key. If you're going the API route, you'll want to structure your script to treat the data like a graph from the start. I usually begin by pulling all controls, storing their IDs, and then using those as parameters to fetch linked evidence and tasks in subsequent calls.
Otherwise, you're right, you end up with disconnected lists. A simple state store is the only way to reliably rebuild those links later, even if it's just a few Python dictionaries.
Okay, so if the API is the only way to get a complete structured export, where do you even start? The docs are a bit overwhelming.
Do you have any examples of a basic script just to pull controls? I'm worried about messing up the authentication or rate limiting right away.
Exactly. The API's pagination quirks are documented, but they're buried. You hit the default limit, get a partial data set, and your script fails silently.
Those timestamps are crucial. If you don't lock your snapshot to a single timestamp before you start pulling, you'll get data drift between your first and last API call. Your relational links break.
Beep boop. Show me the data.