One of the most critical, yet often overlooked, aspects of deploying a reliable agentic workflow is the validation of an agent's output before it commits to an action or passes data to the next step. Without structured validation, you risk agents hallucinating API call parameters, generating malformed JSON, or proceeding with factually incorrect data that cascades through a multi-agent chain.
SuperAGI provides a straightforward mechanism for this through the `Validation` class and its integration into the action's `execute` method. The core idea is to intercept the raw output from the LLM, run it through your validation logic, and only proceed if validation passes. If it fails, you can either retry, modify the input, or halt execution with a clear error. Below is a practical implementation pattern.
First, you define a custom validation class that inherits from `Validation`. This class must implement the `validate` method, which returns a `ValidationResult` object.
```python
from superagi.lib.validator import Validation, ValidationResult
class StructuredJSONValidation(Validation):
def validate(self, llm_response: str, **kwargs) -> ValidationResult:
"""
Validates that the LLM response is a parseable JSON object
containing required keys and that specific fields meet criteria.
"""
import json
try:
data = json.loads(llm_response)
except json.JSONDecodeError as e:
return ValidationResult(
success=False,
error=f"Invalid JSON format: {e}. LLM Response: {llm_response}"
)
# Example: Validate that a 'summary' field exists and is under 500 chars
required_keys = ['summary', 'sentiment']
for key in required_keys:
if key not in data:
return ValidationResult(
success=False,
error=f"Missing required key: {key}"
)
if len(data['summary']) > 500:
return ValidationResult(
success=False,
error="Summary field exceeds 500 character limit."
)
# Validate sentiment is from allowed list
allowed_sentiments = ['POSITIVE', 'NEUTRAL', 'NEGATIVE']
if data['sentiment'] not in allowed_sentiments:
return ValidationResult(
success=False,
error=f"Sentiment must be one of {allowed_sentiments}."
)
# If all checks pass, return success and optionally the sanitized data
return ValidationResult(success=True, value=data)
```
Next, you integrate this validator into your custom action's `execute` method. The key is to call the validator *after* you receive the LLM's response but *before* you use that response for any consequential step (like an API call or writing to a resource).
```python
from superagi.actions.base_action import BaseAction
from superagi.models.agent import Agent
class ValidatedAPICallAction(BaseAction):
def __init__(self, llm, agent_id, memory):
super().__init__(llm, agent_id, memory)
self.validator = StructuredJSONValidation() # Instantiate your validator
def execute(self, input_data: str):
# Step 1: LLM generates a preliminary response (e.g., a JSON string for an API)
prompt = f"""
Given the input: {input_data}
Generate a JSON object with 'summary' (under 500 chars) and 'sentiment' (POSITIVE/NEUTRAL/NEGATIVE).
"""
llm_response = self.get_llm_response(prompt)
# Step 2: Validate the response
validation_result = self.validator.validate(llm_response)
# Step 3: Act based on validation outcome
if not validation_result.success:
# Option A: Retry with corrected prompt, feeding back the error
retry_prompt = f"""
Previous response was invalid: {validation_result.error}
Correct the following: {llm_response}
Provide only the corrected, valid JSON.
"""
corrected_response = self.get_llm_response(retry_prompt)
# Re-validate the corrected response
validation_result = self.validator.validate(corrected_response)
if not validation_result.success:
# If retry also fails, raise a clear error
raise ValueError(f"Validation failed even after retry: {validation_result.error}")
llm_response = corrected_response
validated_data = validation_result.value
else:
validated_data = validation_result.value
# Step 4: Proceed with validated, type-safe data
api_payload = {
"analysis_summary": validated_data['summary'],
"sentiment_score": validated_data['sentiment']
}
# ... make your secure API call here with the validated payload
return f"Successfully processed and validated data: {api_payload}"
```
The trade-offs with this approach are important to consider:
* **Latency vs. Reliability:** Each validation (and potential retry) adds to execution time but drastically reduces the probability of downstream failures.
* **Validation Scope:** Validation logic can be as simple as schema checks or as complex as calling a secondary verification tool or knowledge base. Be mindful of not recreating the primary task within the validator.
* **State Management:** For multi-step agents, you may need to persist validation results or errors in the agent's memory to inform subsequent actions.
This pattern moves you from hoping the LLM output is correct to actively enforcing a contract, which is essential for any production-grade agent system.
brianh