My results after asking Humata the same question 10 times - consistency score inside.

I decided to put Humata through a basic but critical test: consistency. If I can't trust a tool to give me the same answer to the same question, it's useless for any serious CI/CD or documentation workflow.

I uploaded a straightforward Jenkins declarative pipeline file and asked the same question ten separate times in new chats: "Summarize the stages in this Jenkins pipeline and identify the agent used."

The Jenkinsfile was simple:

```groovy
pipeline {
agent {
docker {
image 'maven:3-alpine'
args '-v /tmp:/tmp'
}
}
stages {
stage('Build') {
steps {
sh 'mvn clean compile'
}
}
stage('Test') {
steps {
sh 'mvn test'
}
}
stage('Deploy') {
steps {
sh 'echo "Deploying to staging..."'
}
}
}
}
```

**The Results:**
* **7 out of 10** responses were perfectly consistent: correctly listed the three stages (Build, Test, Deploy) and identified the agent as a Docker agent using the `maven:3-alpine` image.
* **2 out of 10** were mostly correct but had minor formatting differences (e.g., listing stages with dashes vs bullet points).
* **1 out of 10** was a complete miss. It hallucinated an additional "Analysis" stage and claimed the agent was "unspecified," which is plainly wrong.

**Consistency Score: 70%.** For a simple, factual extraction task, that's not great. The 30% flakiness rate is concerning.

The takeaway for our use cases: Humata can be helpful for quick queries, but you **must** verify its answers, especially when deriving actionable steps for pipeline configuration or infrastructure code. You wouldn't accept a 30% failure rate from a linter or a test suite—don't accept it from your documentation assistant either. For precise, repeatable automation contexts, this inconsistency is a major pitfall.