What's the best way to test Claw's claim that it 'never stores prompts'?

Yeah, that hashing/tokenizing angle is something I wouldn't have thought of. Makes a network trace basically useless.

So even a clean DSAR might not be definitive if they stored a hashed version? How would you ever verify they don't have that?

For SOC 2 or GDPR, credible evidence requires a multi-layered audit approach, not just network analysis. You need to combine technical tests with contractual controls. Start by requesting their data flow diagrams and system architecture reviews as part of your vendor assessment. Look for any data stores, queues, or logging systems where prompt data could transit.

Then, as others have noted, amend the Data Processing Agreement to include a specific right-to-audit clause for prompt handling. This lets you verify their logging configurations, S3 bucket policies, or Kafka stream deserialization logic to confirm prompts are excluded. A DSAR is a good operational test, but you must time it after any stated ephemeral retention period, like 5 minutes for in-memory caching, to avoid false positives.

Finally, don't overlook the client-side vector. Use browser developer tools to inspect network requests from their chat widget for calls to third-party analytics or session replay domains. Their privacy policy might claim no storage, but a misconfigured Segment.io script could be sending every user input elsewhere.

Good point about the SOC 2 report. When you look at it, do you focus on a specific control section? I've gotten those reports before and they're long. I wouldn't know where to start to find the controls for data disposal they mention.

Yeah, those reports are massive and the legalese doesn't help. I always get stuck in the "Criteria" section.

I'd look at the **CC6.1** control series first, it's about logical access. That should tell you who *can* access data, but you're right, disposal is trickier. I think that's often under **CC7.1** for system operations, or maybe in the vendor's own supplemental controls.

Has anyone ever gotten a useful answer just by asking the vendor's security team to point you to the specific control numbers for data retention and purging? Or do they just hand-wave back to the whole document?

Asking their security team usually gets you a vague reference to the "Logical Access" section. It's useless.

You need to dig for vendor-specific controls. Look for "Additional Criteria" at the end of the SOC 2. That's where they'd have to document prompt data handling if they have a special process for it. If it's not there, they probably don't have a control.

Even CC7.1 is too generic. It'll say "data is disposed per policy." You need the policy. Request their data retention schedule as a separate artifact, then cross-reference the control IDs they list there back to the report. If they can't produce the schedule, you have your answer.

The DSAR test with a unique prompt is smart, but you have to time it right. I'd wait a full 24 hours before submitting, just in case they have any daily batch processing or backup routines that could temporarily hold data.

Also, don't just ask for their SOC 2. Ask them specifically which control IDs in the report cover the "prompt data" lifecycle. If they can't point to them immediately, that's a red flag. Their claim should be baked into their controls.

You might also try sending a prompt with a clear PII pattern you can search for later, like a fake email address. Monitor any outgoing calls from their widget in your browser's network tab to see if it gets sent elsewhere, like to a third-party analytics endpoint they forgot about.

Yeah, the "Additional Criteria" section is key. I've seen vendors list custom controls there for things like ephemeral queue retention.

But you're right, if it's not documented there, they're probably just leaning on generic CC controls that don't actually prove their specific claim.

One caveat: sometimes the retention schedule is considered confidential. They might push back on sharing it directly, but they should be able to confirm in the DPA that prompt data has a retention period of zero seconds. If they won't put that in writing, game over.

You're right to look for something beyond just trusting the policy. The DSAR test with a unique prompt seems like the most practical first step, as others mentioned.

But what about prompts that contain PII by accident? Say a customer pastes part of an email into the chat, not thinking. If Claw is analyzing content for intent or routing, wouldn't that require some processing that could touch storage, even briefly? I'm not sure their claim would hold in an edge case like that, unless their architecture is truly stateless.

I'm also curious about what happens during an error. If the system crashes while processing a prompt, is there any logging that could capture it? That feels like a loophole in the "never stores" promise.

Agree on DSAR being a strong test, but it's reactive. It only proves they didn't store your *specific* test prompt. Doesn't prove their architecture prevents storage across all prompts.

The third-party analytics callout is critical. I've had to tell devs to strip PII from error logs sent to Sentry three times. It always leaks somewhere.

Exactly right. The DSAR only proves a single data point, not the system. And you hit the core issue with the third party logging.

Even if Claw's main pipeline is clean, I've seen compliance fail because a dev added a debug log to New Relic that includes the full prompt object. Their claim is meaningless unless they audit every single logging statement and external service call in their codebase, including third party libraries. Most shops can't even track that.

You've raised the core compliance challenge: verifying a negative claim about data handling. Credible evidence requires a multi-layered audit, not just a single test. I'd focus on three concrete artifacts.

First, request their technical architecture diagram and data flow map. Look for any storage components (queues, caches, databases) between the ingress point and the LLM inference endpoint. The claim requires a purely ephemeral path.

Second, as others noted, scrutinize the SOC 2's "Additional Criteria" for a vendor-specific control that explicitly defines the prompt lifecycle. If it's missing, the claim rests on generic controls which are insufficient. A retention schedule showing a zero-hour period for prompt data is non-negotiable.

Finally, a practical test: instrument a test session with a browser's network monitor and a unique identifier. Check for any POST calls to external domains besides Claw's primary API endpoint. Pay particular attention to error logging services; a single misconfigured call to Sentry or Datadog would invalidate the entire claim. The architecture must be stateless from the load balancer to the model and back.

Network analysis is a decent starting point, but it's insufficient on its own. A Wireshark capture would only show you encrypted traffic to their API, not what they do with it internally.

Credible evidence is a combination of artifacts, not a single test.
* Their **Data Flow Diagram** must show zero persistent storage between your request and the LLM call.
* Their SOC 2 **Additional Criteria** must contain a specific, audited control for prompt data lifecycle with a zero-second retention period. Generic CC7.1 doesn't cut it.
* A **DSAR test** with a unique token is practical, but only proves a single transaction. You need the architectural proof.

If they balk at providing any of those, their claim is just marketing.

You're right to focus on credible evidence beyond the policy statement. I'd start with a direct request for their Data Flow Diagram and a copy of the SOC 2 report. Ask them to highlight the specific control that enforces the "never stores" claim - if it exists, it should be a custom control in the Additional Criteria.

The DSAR test is a good practical step, but you should pair it with a network check for third-party calls. Even if their core system is clean, data often leaks to external logging or monitoring services. Look for calls to tools like Sentry or Datadog while using the chatbot; if the prompt payload appears there, the claim is already broken.

Network analysis for third party calls is a solid suggestion, but even that misses the real architectural weak point: error handling. I've seen three separate teams build "stateless" pipelines that dutifully sent a 500 error with the full, uncleaned prompt context straight to their observability stack. The SOC 2 report might be clean, but the New Relic logs are a toxic dump.

If they can't show you their error logging specification and prove it's enforced in their CI/CD - and I mean prove it, not just show you a policy - then the claim is functionally meaningless for production use. Stuff *always* fails. That's where the data spills.

That's a fair technical point about asynchronous batching. I hadn't considered cold storage layers.

It raises a question about how they'd handle retries or load balancing. If a request fails, even transiently, wouldn't it need to be queued somewhere to be re-processed? That temporary queue would constitute storage, however brief, which contradicts the "never" claim.