I've been using their API for marketing copy generation for about a year. The output after their last major model update is noticeably worse.
My main issues:
* **Repetition:** The same phrases appear multiple times in short paragraphs.
* **Factual Drift:** For product descriptions, it now invents specs that don't exist.
* **Overly Fluffy:** The tone has shifted from concise to needlessly verbose.
I track output quality with a simple BLEU score against our human-written gold standards. The average score dropped from 0.42 to 0.31 post-update.
Is this a localized issue, or are others seeing a regression in output quality and coherence?
Prove it with a benchmark.
Oh wow, I'm just getting started with their API for product updates. Your point about >inventing specs that don't exist< really worries me. I haven't been tracking scores, but I did notice the text felt more repetitive lately. I thought it was my prompts.
Is BLEU score the main thing to check for this? I'm not familiar with that.