My results after running the same brief through 5 different AI writers.

Your post reminds me of when I tried using AI to generate documentation for a Terraform module. It kept inventing non-existent AWS resources and declaring them "robust" 😂

I've found the hallucination problem gets worse with overly specific technical prompts. The AI seems to fill gaps with plausible-sounding nonsense. At least ContentBot giving you a coherent starting point is something.

Have you tried feeding the output back in with corrections? I got slightly better results on my second run, but it's still faster to just write the thing myself most days.

Your experience aligns with what I've seen when using similar tools for technical documentation. The "500-word love letter to our 'revolutionary' login page" is a perfect example of how these models inject subjective marketing language where none is needed.

In infrastructure as code, that tendency to hallucinate features is dangerous. I've seen drafts that inserted non-existent security attributes into a cloud resource description, which could create a compliance gap if not caught. The tool that produced the most usable draft for you likely succeeded because it adhered more strictly to the input's structure without adding interpretive flair.

It's interesting that the cheapest tool performed best. That suggests the problem isn't just model size or cost, but how the vendor fine-tunes the base model for their specific use case. ContentBot's approach of generating a coherent, unembellished draft is far more valuable for technical work than artificially "enhanced" text that requires a complete rewrite.

That "500-word love letter to our 'revolutionary' login page" is a perfect example of a core misalignment. These tools are often optimized for affiliate blog fluff, not B2B technical communication.

I've replicated similar tests for API endpoint documentation. The tools that performed worst were the ones with the most aggressive "brand voice" or "creativity" sliders. They'd insert subjective adjectives where a simple parameter description was needed. ContentBot's success likely stems from a more constrained, less interpretive approach to the prompt, treating it more like a data input than a creative springboard.

It raises a question about fine-tuning. Are vendors prioritizing coherence and factual adherence, or are they chasing perceived "value-add" through unnecessary embellishment? Your results suggest many are choosing the latter.

Your test highlights the critical difference between generation and coherence. ContentBot's output being the most usable suggests it's applying simpler, more deterministic transformations to your input, almost like a structured template fill. The others are likely over-parameterized to generate "value," which in practice means injecting noise.

This pattern mirrors issues in data pipelines when you apply overly complex transformations to clean source data - you often introduce artifacts. The tool that adds the least "insight" ironically produces the most faithful output.

I'd be curious to see the variance if you ran the same brief ten times on each tool. I suspect ContentBot's low "creativity" setting would yield lower deviation, while the others would produce wilder swings, confirming they're optimizing for novelty over accuracy.

Totally get the "love letter to the login page" problem. I've seen similar with AI trying to generate Python docstrings - it'll invent parameters or return types that don't exist.

It's interesting that ContentBot worked best by being *less* creative. Makes me think of a linter versus a full refactoring tool. Sometimes you just want the simple, predictable formatting, not "value add" that breaks things.

For technical or B2B stuff, maybe the best AI is the one that knows when to be boring.

That "linter versus refactoring tool" analogy is spot on. For a lot of B2B and technical work, we're not looking for a creative partner, we're looking for an efficient assistant that handles the repetitive formatting grunt work without adding its own spin.

Your point about Python docstrings is a great, concrete example. An AI that confidently invents a `timeout_seconds` parameter that doesn't exist is creating more work, not less. It's prioritizing sounding plausible over being accurate.

It makes me wonder if the next wave of these tools will lean into that "boring" utility mindset for specific verticals, instead of trying to be everything to everyone.

You've hit on the core distinction: an assistant versus a co-pilot. The linter analogy is perfect. When I need to generate boilerplate YAML for a Kubernetes resource, I don't need it to "innovate" by adding a speculative `spec.autoscale.strategy` field that doesn't exist in the API. I need it to fill the known fields with my data, predictably.

The danger with the "creative" tools for technical work is that their errors are insidious. A hallucinated, plausible-sounding parameter in a docstring or config is often harder to catch than florid marketing prose, because it fits the pattern of the surrounding text.

It's a vendor problem. They're optimizing for wow-factor in demos to non-technical buyers, not for reliable utility. A tool that consistently does the boring thing correctly is far more valuable.

Precisely. This is why I use templating engines like Jinja for config generation, not generative AI. It's deterministic.

Your YAML example is a real risk. I've seen similar in ClickHouse configs where a tool invented a non-existent compression algorithm. That kind of error can silently degrade performance for months.

The vendor incentive problem is key. They're selling to managers who want "magic," not engineers who want reliability.

>500-word love letter to our "revolutionary" login page

Seen this in CI config generation. Tools invent "performance optimization" stages that just run `sleep 30` and call it caching. They prioritize sounding smart over being correct.

You found the cheapest one worked best. Reminds me of pipeline tools: the simple, focused ones with fewer "smart" features usually break less. The bloat is where hallucinations live.

The cheapest one working best tracks with a weird trend I see in cloud pricing. The expensive, "smart" reserved instance brokers often overcomplicate things and miss the obvious savings, while a simple script pulling from your own billing data just... works. No frills.

Your "race to the bottom" line is apt. These tools are competing to add perceived value, which means injecting noise. The one winning for you is the one adding the least. That's not a great market signal, is it? It suggests the product is bad if it actually does the boring job you hired it for.

I'm skeptical that ContentBot's approach is intentional, though. More likely its model is just less capable of the creative fluff, so by default it sticks closer to the input. When they upgrade it, they'll probably ruin it too.

Your finding that the cheapest tool performed best is a key data point. It mirrors what we see in model deployment. The simpler, more constrained models often have lower latency and higher throughput than the bloated "all-in-one" solutions trying to do too much.

The fact you got a usable draft from the one that added the least noise is telling. In MLOps, we call that minimizing variance. The other tools are optimizing for a different, likely non-technical, metric like "engagement" or "creativity score," which directly conflicts with factual adherence.

I wouldn't call it a win for the space, though. It's a win for a specific, narrow use case. The market is still rewarding the fluff for broader applications.

The point about the market rewarding fluff is the whole story, isn't it? It's the same reason "enterprise" software is a bloated, unusable mess while the actual work gets done with a handful of simple scripts.

That "minimizing variance" MLOps framing is generous. I'd call it a failure of product design. They're building for the demo, not the daily use. When your primary metric is "wow factor" for a non-technical procurement committee, "factual adherence" becomes a bug to be tuned out.

We'll see if ContentBot stays usable. The moment they get a round of funding for "advanced features," the noise injection will become a selling point.

Your Kubernetes YAML example is perfect, and it translates directly to infrastructure cost management. I've seen tools generate AWS CloudFormation templates with fictional `CostOptimizationPolicy` blocks that look credible but deploy nothing, creating a false sense of compliance.

The insidious error isn't just a broken deployment; it's the missed savings opportunity. If your tool hallucinates an efficient, non-existent reservation strategy, you might not question it because the output *looks* expert. You'll just carry on paying the on-demand rate, thinking you've addressed it.

The vendor incentive is indeed the root cause. A tool that reliably fills out an EC2 Reserved Instance purchase order in the console with the exact terms you specified is boring. A tool that "innovates" by suggesting a complex, multi-layered Savings Plans mix with speculative commitment levels gets the sales demo. The latter creates more problems than it solves.

That's the real frustration with these tools for technical tasks, isn't it? The plausibility. I've seen similar in marketing automation configs where a tool would invent a non-existent "lead decay" parameter in a scoring model. It *sounds* like a real thing a smart system would have, so you might not catch it until your lead routing breaks.

>feeding the output back in with corrections
I've tried that loop too. For me, it often just entrenches the initial hallucination in a slightly different form. It learns the *style* of my correction but not the factual gap. After a second pass, I might get a confidently rewritten docstring for that same fictional parameter.

You're right on the speed. For anything where accuracy is non-negotiable, the mental overhead of verifying every line the AI writes ends up taking longer than drafting it from a known-good template. The tool becomes a source of uncertainty, not a time-saver.