Alright folks, I need to vent a bit and see if I'm the only one experiencing this. I've been using Writesonic (and Chatsonic) for several months now, primarily to help draft documentation and procedural guides for my IaC projects. Like many of you, I was thrilled when they introduced the "factual" mode in Chatsonic. The promise of fewer hallucinations and more reliable, verified information was a game-changer for technical writing.
However, after putting it through its paces on topics I'm an expert in—like Ansible playbook structure, Terraform module best practices, and Vault secret engine configurations—I've found the "factual" mode to be, frankly, not factual at all. It's confidently incorrect in subtle ways that a beginner wouldn't catch, which is dangerous.
Here's a concrete example from last week. I asked in factual mode:
> "Provide a secure Ansible playbook snippet to create a restricted sudo user on Ubuntu 22.04, using the `ansible.builtin.user` and `ansible.builtin.lineinfile` modules."
The output *looked* professional and had all the right keywords. But it contained critical flaws:
```yaml
- name: Create restricted sudo user
ansible.builtin.user:
name: "{{ new_user }}"
password: "{{ 'Password123!' | password_hash('sha512') }}" # Problem 1
groups: sudo
append: yes
- name: Allow user to run only specific commands via sudo
ansible.builtin.lineinfile:
path: /etc/sudoers.d/{{ new_user }}
line: "{{ new_user }} ALL=(ALL) /usr/bin/systemctl restart nginx" # Problem 2
create: yes
validate: "/usr/sbin/visudo -cf %s"
```
**The issues:**
1. **Hardcoded Password:** It used a literal plaintext password (`Password123!`) in the playbook, which is a massive security anti-pattern. The correct approach is to use `vars_prompt` or retrieve the hash from a vault.
2. **Incorrect Sudoers Syntax:** The line `{{ new_user }} ALL=(ALL) /usr/bin/systemctl restart nginx` is wrong. The `(ALL)` indicates running as *any* user, not *any* command. The format for command restriction should be `{{ new_user }} ALL=(root) /usr/bin/systemctl restart nginx`. This is a subtle syntax error that could lead to over-privileging.
This is just one instance. I've seen it hallucinate Terraform `aws_s3_bucket` argument names and misstate the order of operations for Vault's transit engine. The "factual" flag seems to just make the model *sound* more authoritative, not actually ground its responses in verified data.
I've resorted to a new workflow: using Chatsonic as a *first draft* engine, then applying my own knowledge and external documentation to vet every single line. It's still a time-saver, but it's not the trusted "factual" assistant I hoped for.
Has anyone else done a deep dive on this with technical topics? What's your verification process? I'm starting to think we need to treat all LLM output, regardless of "modes," as potentially flawed code that requires peer review.
—John
Keep it simple.