Just spent an hour untangling a mess caused by an AI assistant confidently hallucinating details about a real, public API. It's a perfect example of why you can't trust these tools for vendor-specific implementation without verification.
I was evaluating a workflow automation between **HubSpot** and **Airtable** for a client. I asked a popular coding assistant: *"Show me the Python code to create a new contact in HubSpot using the API, including the required OAuth 2.0 header and a sample JSON payload for a basic contact."*
The assistant gave me back code that looked plausible, with a `requests.post` call. The problem? It fabricated a required field.
**The Hallucinated Output:**
The assistant stated that the `properties` object in the JSON **must** include a field named `"lifecyclestage"` with a value like `"lead"` or `"subscriber"`, and that omitting it would cause a 400 error. It presented this as a firm requirement.
**The Reality:**
I tested it. The `lifecyclestage` property is **optional** in the HubSpot Contacts API. The actual only truly required property for a basic create is `"email"`. The fabricated "requirement" would have sent me down a rabbit hole of unnecessary logic for setting a default lifecycle stage. The correct minimal payload is:
```json
{
"properties": {
"email": "test@example.com"
}
}
```
**How to Catch This:**
* **Always cross-check with the official source.** I went straight to [HubSpot's API docs]( https://developers.hubspot.com/docs/api/crm/contacts) and found the real schema in seconds.
* **Test the simplest possible case.** If the assistant says something is required, try the call without it. A 400 error will tell you; a 201 Created proves it's wrong.
* **Benchmark against known tools.** I asked the same question to another, more code-specialized AI tool. It gave me the correct, simpler payload, which immediately flagged the first answer as suspicious.
This is a subtle but dangerous failure. It didn't invent a fake endpoint—it invented a fake *constraint* on a real one, which can waste huge amounts of time and add unnecessary complexity. Has anyone else run into similar issues with AI assistants making up SaaS API rules, especially around "required" fields? What's your go-to method for a quick sanity check?
Cheers, Carla
Benchmarking my way to better decisions
And this is exactly why I never trust AI-generated estimates for cloud costs. It's the same pattern - confident, plausible, and dangerously wrong. I've seen it spit out Savings Plan recommendations that would actually cost more than on-demand instances because it hallucinated the discount structure.
You have to verify everything against the source. For an API, that's the actual vendor documentation. For cloud spend, that's the pricing calculator or, better yet, a real bill screenshot. Never implement a "cost optimization" it suggests without checking the math yourself.
show me the bill