As an integration consultant who routinely consumes threat intelligence feeds for client security dashboards, I've been evaluating the Recorded Future daily digest for automated ingestion into SIEM and SOAR platforms. A persistent challenge I'm encountering is the signal-to-noise ratio, specifically the volume of low-confidence indicators that trigger unnecessary workflow executions.
My objective is to filter the digest content *before* it reaches our middleware layer (e.g., Workato, Zapier, or a custom webhook processor). I want to apply a confidence threshold so that only indicators meeting a specific confidence level are forwarded for processing. From my analysis of the API and portal, I understand confidence levels are intrinsic to the data, but the daily digest email itself appears to be a monolithic delivery.
My primary questions for the community are:
* **Source-Side Filtering:** Is there a method to configure the daily digest subscription within the Recorded Future portal to exclude low-confidence indicators (e.g., "Low" or "Moderate") at the point of generation? I have explored the digest settings but found no such granularity.
* **API-First Approach:** If portal filtering is not possible, what is the most effective method to programmatically fetch the digest content and apply filters? I presume this involves:
* Calling the Recorded Future API to retrieve the relevant intelligence that would be in the digest.
* Applying a confidence filter (e.g., `confidence: "High" OR confidence: "Very High"`) to the query or the result set.
* Having my middleware act on this filtered stream instead of the email.
A conceptual code block for the filtering logic in a Node.js middleware function would look like this:
```javascript
// Pseudo-code for processing API-derived indicators
async function processFilteredIndicators() {
const intelligenceItems = await recordedFutureAPI.getDailyDigestItems();
const highConfidenceItems = intelligenceItems.filter(item => {
return item.confidence === 'High' || item.confidence === 'Very High';
});
// Proceed with integration only for high-confidence items
if (highConfidenceItems.length > 0) {
await forwardToSiem(highConfidenceItems);
await createSoarIncidents(highConfidenceItems);
}
}
```
* **Workflow Impact:** Has anyone designed a similar filtering pipeline? I am particularly interested in any pitfalls regarding missed context when removing lower-confidence items, or if there is a recommended practice for handling them in a separate, low-priority queue.
I am seeking concrete implementation experiences rather than theoretical advice. Any insights into API endpoints, webhook configuration parameters, or middleware connector configurations (specifically for Workato or Zapier) would be greatly appreciated.
- Mike
- Mike
You're correct that the portal's digest settings lack granular confidence filtering. I've worked on similar integrations, and the email digest is fundamentally a convenience output, not a configurable data pipeline.
Your API-first approach is the only viable path for pre-filtering. You'd need to bypass the digest email entirely and replace it with a scheduled script that calls the relevant `/alert` or `/intelligence` endpoints, applying a `confidence` parameter in your query. For instance, using the alerts endpoint with `?triggered=[timeframe]&confidence>=80` would replicate the digest concept but with your threshold baked in.
This introduces operational overhead, of course. You now own the polling schedule, error handling, and the data transformation into your middleware's expected format. The trade-off is direct control over the data quality entering your automation workflows. Have you calculated the potential reduction in unnecessary workflow executions against the development and maintenance cost of this custom pipeline?
Show me the numbers, not the roadmap.
Yep, the API workaround is the only real option. But you're glossing over the biggest hidden cost.
> Have you calculated the potential reduction in unnecessary workflow executions against the development and maintenance cost of this custom pipeline?
You can't calculate that reduction without historical data on confidence scores per digest item, which they likely don't have logged. You're building a pipeline to *get* the data needed to justify building the pipeline.
And now you're on the hook for the script's compute costs, monitoring, and the API calls themselves. If the vendor changes a field name or endpoint, your pipeline breaks and you're back to noise until you notice. That's the real trade-off.
show me the bill