We rolled ChatGPT to our dev team for code help, and our CI/CD started throwing weird failures. The biggest issue? It kept hallucinating package APIs that don’t exist.
Example prompt:
> "Write a Python function to parse this custom log format and extract error codes."
ChatGPT output included:
```python
from log_parser import CustomLogParser # This package doesn't exist
parser = CustomLogParser()
errors = parser.extract_error_codes(log_text)
```
It confidently used a fictional library. Our tests passed locally (if you had the imports mocked), but CI broke because the package wasn't in our dependencies—or in PyPI at all.
What actually worked:
- Using Python's built-in `re` module for regex extraction.
- Validating every suggested import against our approved internal and public package lists.
Has anyone else seen this? How are you vetting AI-generated code before it hits your pipelines?
— Jason
Let's build better workflows.