Check out what I made: A/B testing ad creatives using 30 DALL-E 3 variants

In our ongoing optimization of digital marketing assets, we hypothesized that generative AI could systematically produce a more diverse and effective set of ad creatives than traditional manual design for A/B (or multivariate) testing. This post details a controlled experiment to validate that hypothesis using DALL-E 3 via the OpenAI API, generating 30 distinct image variants for a single product advertisement.

The subject was a hypothetical new eco-friendly water bottle. The core prompt engineering strategy involved creating a modular template, isolating specific variables for systematic variation. The base prompt structure was:
```
Professional product photography of a [COLOR] reusable water bottle made of recycled materials, placed on a [SURFACE] in a [SETTING]. The style is [STYLE]. The lighting is [LIGHTING].
```
The variables were defined as arrays:
- COLOR: matte black, ocean blue, forest green, terracotta, clear frost
- SURFACE: weathered wooden desk, mossy stone, modern glass table, mountain rock
- SETTING: cozy home office, lush garden, minimalist studio, mountain summit at dawn
- STYLE: hyperrealistic, muted pastel illustration, bold graphic design, soft focus
- LIGHTING: soft morning light, dramatic side lighting, even studio light

A script was written to perform a Cartesian product of these arrays, though not all 320 possible combinations were generated; 30 were selected to ensure coverage across all variable categories while maintaining a manageable batch size for evaluation. Each generated image was then paired with three different headline variants, creating 90 total ad units for the testing platform.

Initial performance data from the first wave of the test, measured over 48 hours and approximately 200,000 impressions, indicates a significant spread in engagement metrics. The top-performing 10% of image variants, which were not clustered around a single variable set but included examples from 'muted pastel illustration' and 'hyperrealistic' styles, achieved a click-through rate (CTR) 2.3 standard deviations above the mean of our historical, human-designed benchmarks. The latency and cost of generation were non-trivial but calculable. The API call pattern for 30 images introduced a predictable bottleneck, with a mean generation time of 12.4 seconds per image and a total cost of $6.00 for the set at standard resolution, which is orders of magnitude below typical freelance design costs for a comparable volume.

This methodology presents a compelling case for the integration of structured AI image generation into creative testing pipelines. The key finding is not that AI art is superior, but that its capacity for rapid, systematic exploration of a high-dimensional creative space (color, texture, setting, style) allows for a more statistically rigorous identification of high-performing visual motifs. The next phase will involve implementing this process into a CI/CD-like workflow, where performance data from one test cycle automatically informs the prompt parameter weighting for the next batch generation, creating a closed-loop optimization system.