I have been conducting a systematic evaluation of generative AI services for text-in, image-out workflows, and Adobe Firefly's text rendering has emerged as a significant outlier—and not in a positive sense. While my primary benchmarks focus on latency and cost, output quality is a fundamental parameter. In the case of Firefly (specifically testing Firefly Image 2 and 3 models via the web interface and the Adobe Firefly API), the generation of text *within images* is consistently and catastrophically poor, rendering the feature unusable for any professional application requiring legible typography.
My standardized test protocol involves generating a series of 20 images per model with specific, clear prompts requesting text. The prompts are designed to be unambiguous, such as:
- "A neon sign that says 'OPEN 24 HOURS' on a dark brick wall"
- "A clean, modern logo for a coffee shop called 'BEAN THERE'"
- "A vintage poster with the text 'TO THE MOON AND BACK' in bold serif font"
The results are remarkably consistent across hundreds of generations:
- **Character Fusion:** Letters are merged into unrecognizable glyphs (e.g., "OPEN" appears as a single blob of light).
- **Random Character Insertion:** Extra letters and symbols appear that are not in the prompt (e.g., "BEAN THERE" might become "BEAQN T/HERE").
- **Complete Gibberish:** In many cases, the output is not even a distorted version of the requested text, but entirely random strings of characters (e.g., "K@L!mN P0").
- **Failure to Isolate:** The model seems unable to treat the requested text string as a discrete, renderable entity separate from the texture of the scene.
This is not a minor quality fluctuation; it is a fundamental failure mode. For comparison, I ran identical prompts through DALL-E 3 and Midjourney's in-painting capabilities. While not perfect, these models produce legible text a significant percentage of the time. Firefly's performance here is near zero.
I am seeking to understand if this is a documented limitation of the model architecture or if there are specific prompting techniques or parameter adjustments (e.g., using the "Stylized Text" feature in the web UI, or adjusting `content_classification` and `styles` via the API) that can mitigate this issue. My current API call structure is:
```python
response = firefly.generate(
prompt="A neon sign that says 'OPEN 24 HOURS'",
width=1024,
height=768,
content_classification="photo",
number_of_images=1
)
```
Has anyone in the community conducted similar stress tests on Firefly's text generation and found a reliable workaround, or is the consensus that this capability is effectively non-functional? For my benchmark reports, I will be forced to categorize it as such unless a reproducible solution is found.
benchmarks or bust