Shared my dashboard config for tracking insider threat indicators.

While ThreatConnect's primary domain is external threat intelligence, its flexible data model and robust API make it a compelling platform for internal security monitoring. I've recently architected a pipeline to ingest, correlate, and visualize insider threat indicators, moving beyond the traditional use case. The core challenge was synthesizing high-volume, disparate internal logs (authentication, data access, endpoint) with the structured intelligence framework ThreatConnect provides.

My solution involves a multi-stage ETL process. First, raw logs from various sources are normalized and key fields (user, resource, timestamp, activity type, risk score) are extracted using a Python-based ingestion job. This transformed data is then posted to ThreatConnect as "Indicators" (type: `Event`) via the API, with observables attached representing the user and asset. Crucially, we use Groups and Tags within ThreatConnect to establish relationships and apply classification labels (e.g., `potential_data_exfiltration`, `after_hours_access_spike`).

The following configuration snippet for our Airbyte connector (custom, using the Python CDK) shows how we structure the API call to create these Event indicators:

```yaml
source_config:
base_url: "https://.threatconnect.com"
api_access_id: "${SECRET_TC_ID}"
api_secret_key: "${SECRET_TC_KEY}"
indicator_type: "Event"
body_template: |
{
"name": "Insider Event: {{ user }} - {{ activity }}",
"eventDate": "{{ timestamp }}",
"tags": ["insider_threat", "{{ calculated_risk_tier }}"],
"attributes": [
{"type": "Description", "value": "{{ raw_log_summary }}"},
{"type": "External ID", "value": "{{ internal_correlation_id }}"}
],
"observables": [
{"type": "User", "value": "{{ user_identifier }}"},
{"type": "Host", "value": "{{ asset_hostname }}"}
]
}
```

The true analytical power is unlocked in the aggregation and reporting layer. I use dbt to model the data extracted from ThreatConnect's API (via a scheduled export to BigQuery). This allows for complex joins with our internal HR and asset management databases to enrich context. The final dashboard, built on Looker, tracks several key metrics over time:

* Rate of high-risk `Event` creation, grouped by organizational unit.
* Correlation between tagged incidents and subsequent confirmed security events.
* Mean time from indicator ingestion to analyst review (measuring workflow efficiency).

This approach transforms ThreatConnect from a purely external intelligence repository into a unified security operations ledger. The pitfalls to note are primarily around scale—high-frequency internal logging can generate order-of-magnitude more indicators than typical threat intel feeds, necessitating careful API rate limit handling and a thoughtful tagging taxonomy to avoid alert fatigue. I'm keen to hear if others have adapted TC for internal use cases and how you've managed the data pipeline complexities.