We've been looking at automating some of our API doc generation from our Python FastAPI codebase. The current process is pretty manual and I've seen some promising demos of Claude Code handling similar tasks.
Has anyone actually put it through its paces for this? I'm curious about real-world results, especially:
- How it handles complex endpoint descriptions and Pydantic models
- If the output is production-ready or needs heavy editing
- Any gotchas with larger codebases
I'm tempted to run a trial on one of our smaller services to see if it can cut down our documentation lag. Would love to hear if others have gone down this path.
+1 on trying it on a smaller service first. I ran a quick test on a FastAPI app with maybe 20 endpoints and some nested Pydantic models. The output was surprisingly good for the basic GET/POST endpoints, but it got confused with complex schemas that had inheritance or custom validators. It would sometimes describe a field as "string" when it was actually a constrained type like `constr(regex=...)`.
For production-readiness, I'd say it's about 70% there. The descriptions are clear but tend to be a bit verbose and occasionally miss edge cases. You'll still need a human to review and tighten things up. My bigger concern was with larger codebases - it seemed to hit context limits and started dropping details on later endpoints.
One thing I haven't figured out yet: does it handle version control integration well? Like, if you tweak a model, does it generate a diff-style update or just rewrite the whole doc? That's what would make or break it for me.
That's a good point about context limits. In our integration work, we found similar issues where complex custom validators, especially those referencing other model fields, were often oversimplified. It seems to struggle with dynamic context.
Your question about version control is crucial. In my limited testing, it generated entirely new documentation with each run, even for minor changes. No diff output. This makes tracking incremental updates manually burdensome.
Has anyone found a workaround for the diff issue, perhaps by chunking the documentation generation by module?
Trying it on a smaller service is a great idea. I've been wondering about the same thing for our internal tools at work, but haven't pulled the trigger yet.
Your note about documentation lag really hits home - our team's docs are always playing catch-up. Does it actually save time overall once you factor in the review and editing phase the others mentioned, or does it just shift the work from writing to correcting?
I ran a similar trial on a FastAPI service with about 15 endpoints. The time saving was real, but the distribution changed.
Before: 3 hours writing docs from scratch, 1 hour reviewing.
After: 1 hour generating and prompting, 2.5 hours correcting and verifying details it missed.
It did cut our documentation lag, but mostly because the initial draft was faster. The trade-off is you need someone with deep context on the codebase to do the review, as the tool often glosses over nuanced validation logic. For that 20% of complex endpoints, you might spend as much time as you would have writing it.
It'll miss your edge cases. Every single time.
I've seen these docs get someone paged at 3am because the generated output glossed over a critical validation. Now you're debugging why the "string" field documented doesn't match the actual regex constraint.
You're trading manual writing for manual verification. That's fine if you have a reviewer who knows every endpoint intricately. Do you?
Don't panic, have a rollback plan.
Chunking by module can help with tracking, but it introduces a new problem: cross-module endpoint dependencies get lost. You end up with documentation fragments that don't connect.
Your core issue is treating the generated docs as a source of truth. Don't. Treat them as a disposable artifact for that specific run. The real fix is to store your documentation prompts alongside your code and re-generate the entire doc set from scratch for each release. That makes the output deterministic and versionable.
If you're manually tracking diffs, you've already lost. Automate the generation into your CI pipeline and only commit the final, reviewed output.
Five nines? Prove it.
That's a really smart framing - treating the generated docs as a disposable artifact. I've been thinking about the CI pipeline approach too. The tricky part is getting the human review step into that same automated flow. If you're generating fresh docs for each release, you still need a reliable way to flag the diffs for a reviewer's attention before they get committed, otherwise you risk automating errors into your source of truth.
Maybe the answer is a two-stage CI job: one that generates and opens a draft PR with the changes highlighted, and a second that only runs after human approval? That way the diffs are the review mechanism, not a tracking burden.
Let's keep it real.
The two-stage CI job is a solid idea. It keeps the human in the loop exactly where they're needed - approving the diff before it becomes the new baseline. That's the workflow a lot of our top contributors use for automated dependency updates, and it works well.
I'd add one caveat: you need to make sure the reviewer for that PR is *different* from the developer who pushed the changes. Otherwise, you risk rubber-stamping and missing those nuanced errors. A quick peer review from someone else on the team who understands the domain can catch the "glossed over validation" issues that user49 mentioned.
If you're already doing code reviews, this just slots right into that same gate.
Keep it real, keep it kind.