Skip to content
Notifications
Clear all

Real experience using Freeplay with AWS Bedrock and Claude

2 Posts
2 Users
0 Reactions
67 Views
(@danielk)
Honorable Member
Joined: 4 months ago
Posts: 382
Topic starter   [#16914]

Just finished a proof-of-concept integrating Freeplay with our Bedrock/Claude pipeline. It works, but the setup has sharp edges.

**Key use-case:** We needed to log, trace, and evaluate prompts/completions for a customer-facing agent using Bedrock's Claude models. Freeplay's eval framework is the main draw.

**The integration:** The Bedrock side uses the Freeplay SDK's `patch_boto3` method. It wraps the Bedrock runtime client to auto-capture traces.

```python
import freeplay
freeplay.patch_boto3()

# Your existing Bedrock client calls are now instrumented
client = boto3.client('bedrock-runtime')
response = client.invoke_model(...)
```

**Gotchas:**
* The patching is global. Can cause side-effects if you have other Bedrock clients that shouldn't be logged.
* Bedrock's response format is low-level. You need to extract the actual completion from the response body. Freeplay's examples don't always make this clear.
* Evaluations on Bedrock traces require careful prompt template mapping, as the raw Bedrock payload is JSON, not a simple text prompt.

**Verdict:** Useful for centralized tracing and running evals against Bedrock traffic. The patching feels invasive, but it's effective. Would only recommend if you're heavily invested in Claude via Bedrock and need the evaluation features. For simple logging, you might just build your own wrapper.

-dk


Trust but verify, then don't trust.


   
Quote
(@jasonl)
Eminent Member
Joined: 4 months ago
Posts: 23
 

The global patching is indeed a significant constraint. We've mitigated this in our setup by creating a dedicated Bedrock runtime client immediately after the patch, and then only using that specific client for the instrumented calls. It's a bit of a manual sandbox, but it isolates the side effects.

Your point about prompt template mapping is crucial. The raw Bedrock request often contains a JSON schema or a complex messages array, which doesn't cleanly map to Freeplay's template variables. We've had to write adapter layers that serialize the specific prompt structure we send to Bedrock into a flat string keyed to a template variable like `{formatted_messages}`. It adds another piece of pipeline to maintain.

How are you handling the extraction of the completion from the response body? We found the need to parse the `response['body'].read().decode()` and then navigate the JSON structure to be a frequent source of errors in our initial logs.


Data beats opinions.


   
ReplyQuote