Anyone tried both Profound and AirOps? Honest comparison

AI Writing Tool Comparisons

Last Post by auditlog 6 days ago

1 Posts

1 Users

0 Reactions

4 Views

RSS

auditlog

(@auditlog)

Estimable Member

Joined: 3 months ago

Posts: 130

Topic starter 19/07/2026 8:18 am [#13952]

I've been tasked with evaluating AI writing tools for generating our internal audit reports and compliance documentation (think SOX control narratives, data processing summaries for GDPR Article 30 records, etc.). The key requirement is consistency, factual accuracy, and the ability to follow a strict template. I've run an identical, detailed prompt through both **Profound** and **AirOps** to see how they handle a real-world, compliance-sensitive task. My method was to treat them like a junior analyst: give them the raw data and a structure, and see what needs correction.

My test prompt was quite specific:

```
Act as a senior security analyst. Using the following raw log entries from our API gateway, produce a concise incident summary for the quarterly audit review. The summary must follow this exact structure:
1. Incident Overview: Date, time, affected resource.
2. Anomalous Activity Description: Concise bullet points.
3. Root Cause Analysis: Based on the log patterns.
4. Remediation Actions Taken: Bullet points.

Here are the raw, unedited outputs.

**Profound Output:**
```
Incident Summary for Quarterly Audit Review

1. Incident Overview:
- Date: January 15, 2024
- Time: Approximately 03:22:17 UTC to 03:23:05 UTC
- Affected Resource: User privileges modification endpoint (/v1/user/privileges) on the production API gateway.

2. Anomalous Activity Description:
- A series of rapid PUT requests to the user privileges endpoint originating from source IP 198.51.100.22.
- Initial requests (03:22:17, 03:22:19) resulted in 401 (Unauthorized) statuses.
- A subsequent request (03:22:21) resulted in a 403 (Forbidden) status.
- After a brief pause, a final request (03:22:25) from a different IP (203.0.113.45) to an admin endpoint succeeded with a 200.
- The original IP (198.51.100.22) then successfully executed a PUT request to the privileges endpoint at 03:23:05, receiving a 200 status.

3. Root Cause Analysis:
- The pattern suggests a potential credential stuffing or brute-force attempt that initially failed.
- The successful admin request from a separate, internal IP at 03:22:25 may indicate a compromised internal account or token was used to elevate privileges, enabling the subsequent successful privilege modification by the original external IP.
- The non-standard user agent string further points to automated, non-browser tooling.

4. Remediation Actions Taken:
- The source IP 198.51.100.22 has been added to the gateway blocklist for all privileged endpoints.
- The internal service account used from IP 203.0.113.45 has been rotated and its permissions are under review.
- Alerting thresholds for rapid 401/403 sequences on privilege endpoints have been tightened.
```

**AirOps Output:**
```
**Quarterly Audit Review - Security Incident Summary**

**Incident Overview**
- Date & Time: January 15, 2024, from 03:22:17 UTC to 03:23:05 UTC.
- Affected Resource: Production API Gateway, specifically the `/v1/user/privileges` PUT endpoint.

**Anomalous Activity**
We observed sequential PUT requests to a sensitive endpoint from an external IP (198.51.100.22). The activity showed failed authentication followed by a successful permission change. Concurrently, there was an admin user list access from a separate IP.

**Root Cause**
The most likely root cause is a multi-stage attack. An external actor attempted to modify user privileges, failed, then an internal system (203.0.113.45) performed an admin action, which may have facilitated the external actor's final successful request. This could involve token leakage or internal account compromise.

**Remediation**
- Blocked IP 198.51.100.22 at the network perimeter.
- Initiated review of internal tools and service accounts.
- Recommended implementing stricter rate limiting on `/v1/user/privileges`.
```

**My Honest Editing Notes:**

What needed fixing in **Profound's** output:
* The timeline logic in the "Anomalous Activity" section is slightly off. It states the internal IP request happened at 03:22:25, but the log shows it is a `GET /v1/admin/users`, not directly a privileges modification. The output implies a causal link ("enabling the subsequent...") that, while plausible, is an interpretation beyond the pure log facts. For audit purposes, I had to reword this to be strictly factual: "Following the 403 error, a contemporaneous successful `GET` request to an admin endpoint was observed from a different IP. Subsequently, the original IP succeeded."
* The "Remediation" bullet points are written as if actions *were* taken. In reality, this was a test. I had to change all verbs to past conditional ("would have been added," "would have been rotated").

What needed fixing in **AirOps'** output:
* More significant issues. The structure deviated from my explicit instruction. I requested four numbered sections; AirOps used bold headers and omitted the numbering.
* The "Anomalous Activity" section is too vague and conflates events. "Concurrently, there was an admin user list access" is inaccurate; the admin `GET` was not strictly concurrent with the final successful `PUT`, and it was a user list access, not just a generic "admin action." This required complete rewriting for precision.
* The "Remediation" section includes a recommendation ("Recommended implementing...") which was not part of the requested "Remediation Actions Taken" structure. This is a speculative addition.
* The language is less formal and less detailed throughout, requiring substantial expansion to meet audit documentation standards.

**Conclusion for my use case:**
Profound adhered more rigidly to the required format and produced a more factually precise, log-centric narrative that required only minor factual adjustments. AirOps took more editorial liberty, made broader inferences, and didn't follow my structural template as closely, which would result in more editing time for compliance-sensitive material. For generating drafts of audit trail summaries, Profound's output provided a superior starting point. However, for a less rigid brainstorming session about potential security scenarios, AirOps' inferred narrative might spark useful investigation lines.

Logs don't lie.

Quote

Topic Tags