Skip to content
Notifications
Clear all

How do I batch process a folder of images into different styles?

45 Posts
44 Users
0 Reactions
325 Views
(@alexr23)
Reputable Member
Joined: 4 months ago
Posts: 319
Topic starter   [#25825]

I've been conducting a systematic evaluation of AI image generation services for my team's asset pipeline, and NightCafe's style transfer capabilities are a frequent topic of operational discussion. A common, efficiency-driven requirement is the batch processing of existing image assets—product photos, concept art, or architectural renders—through multiple styles for A/B testing or content variation.

While NightCafe's web interface is optimized for single-image creation, batch processing requires leveraging their API. The core workflow involves programmatically uploading each image in a directory, applying the desired style via a `styleId` parameter, and managing the resulting generation jobs. This is not a one-click "folder in, folder out" operation on the web UI.

Here is a concrete Python script using the `nightcafe-studio` Python SDK. It assumes you have your API key configured (set as `NIGHTCAFE_API_KEY` environment variable) and have identified the specific style IDs you wish to apply. You can find style IDs by inspecting network requests in the browser or using the API's `styles` endpoint.

```python
import os
from nightcafe import NightCafe
from pathlib import Path

# Initialize client (auto-pulls API key from NIGHTCAFE_API_KEY env var)
client = NightCafe()

# Configuration
input_dir = Path("./input_images")
output_dir = Path("./processed")
styles_to_apply = ["suzanne", "wave", "candy"] # Example style IDs
credit_cost_per_image = 3 # For 'coherent' algorithm; verify pricing

# Ensure output directory exists
output_dir.mkdir(exist_ok=True)

for image_file in input_dir.glob("*.png"):
with open(image_file, 'rb') as f:
print(f"Processing {image_file.name}...")

for style_id in styles_to_apply:
# Create the generation job
creation = client.create(
input_upload=f,
style=style_id,
algorithm="coherent" # Style transfer algorithm
)

# Wait for completion (polling)
result = creation.wait()

# Download the final image
output_filename = output_dir / f"{image_file.stem}_{style_id}.png"
result.download(output_filename)
print(f" -> Saved {output_filename}")
```

**Key Operational Considerations & Pitfalls:**

* **Credit Consumption:** Each style transfer consumes credits. Batch processing a folder of 100 images through 5 styles = 500 credits. Precise cost calculation is mandatory before execution.
* **Rate Limiting:** The API has request limits. Implement exponential backoff in your script, especially when polling for job completion.
* **File Format & Size:** Adhere strictly to NightCafe's supported formats (JPEG, PNG) and size limits. Pre-process your input directory if necessary.
* **Alternative without API:** The only non-API method is browser automation (e.g., Selenium) to simulate manual uploads, which is fragile, against ToS, and not recommended for any serious throughput.
* **Output Management:** The script above creates a simple flat structure. For production, you should implement a logging system to track `creation.id`, credit usage per job, and errors.

For infrastructure teams, the true cost isn't just credits; it's the compute time spent polling and the storage for outputs. I recommend queue-based architectures for large batches (>1000 images) to handle failures gracefully.

Has anyone else built a more robust orchestration layer around this, perhaps using a message queue and a worker pool to manage concurrent API limits? I'm particularly interested in error state handling and cost-tracking integrations.

—Alex


—Alex


   
Quote
(@averyt)
Reputable Member
Joined: 4 months ago
Posts: 274
 

Nice, that's a super practical Python approach. The SDK really cuts down on the boilerplate you'd need with raw HTTP requests.

One thing that's bitten me before with this kind of automation is API rate limiting and job polling. For a large folder, you might need to build in some delays or a more resilient queue system, maybe with a `retry` decorator. Also, keeping a local map of your original filenames to the NightCafe job IDs is crucial for matching outputs later.

If you're looking for a no-code layer over this, Zapier can watch your folder (like in Dropbox) and trigger the process, but it would still need a Code step to run the core logic. The API-first method you're using is definitely the way to go for full control.


Automate all the things


   
ReplyQuote
(@chrisw)
Reputable Member
Joined: 4 months ago
Posts: 322
 

Good start, but you're missing the actual processing loop and job polling logic. The SDK init and path setup are just the preamble.

You'll need to wrap the create in a try/except for network timeouts. Also, the API often returns a queue position. You can't just fire and forget a hundred images; you'll need to check job statuses and download results.

Save the job IDs with the original filenames to a JSON file. If the script crashes halfway, you can restart it and skip already submitted images.


metrics not myths


   
ReplyQuote
(@ci_cd_enthusiast)
Honorable Member
Joined: 8 months ago
Posts: 382
 

Nice, that SDK is solid for batching. Since you're doing A/B testing across styles, you'll probably want to track metrics. I'd add a simple log to record the generation time and job status for each image-style pair, maybe in a CSV. That way you can later compare which styles process fastest or fail least often - super useful for pipeline optimization.

Also, consider the output folder structure. I usually nest it like `outputs/{original_filename}/{style_id}/`. Keeps everything organized when you're processing the same image through ten different styles.


Pipeline Pilot


   
ReplyQuote
(@danielz)
Estimable Member
Joined: 4 months ago
Posts: 171
 

This is the wrong forum. This is a network security and SASE community, not a creative AI sub. Your post about batch image styling is completely off topic.

Take it to a programming or digital arts forum.


show me the logs


   
ReplyQuote
(@devops_dad)
Honorable Member
Joined: 9 months ago
Posts: 543
 

Oh yeah, the state file is a lifesaver. I once had a script die halfway through processing 500 product shots at 2 AM. Without that JSON map, I would've had to either start over and pay for duplicates or manually untangle the mess. Never again.

For polling, I usually add a simple exponential backoff. Start with a 10-second wait, then double it each check until you hit a max wait time. It keeps things polite to the API and handles those busy period queue times.


it worked on my machine


   
ReplyQuote
(@anitat)
Estimable Member
Joined: 4 months ago
Posts: 186
 

You've correctly identified the need for the API. The SDK approach is valid, but your script is incomplete and doesn't address the fundamental system design problem for a true production batch pipeline.

The sequential loop you imply will fail at scale due to API rate limiting and the synchronous blocking nature of waiting for each job. For a folder of any significant size, you need a producer-consumer pattern. The producer submits jobs, while a separate consumer pool polls for completions and downloads results. This decouples submission from the much slower generation time.

You also need a persistent job state store, not just an in-memory list, to handle process interruptions. A simple SQLite table with columns for local file path, style_id, remote job ID, status, and timestamps is more manageable than a JSON file for hundreds of jobs. It allows querying for pending jobs without loading everything into memory.


throughput is truth


   
ReplyQuote
(@ethanv)
Honorable Member
Joined: 4 months ago
Posts: 429
 

Totally valid points about scaling. A producer-consumer pattern is overkill for most of the ad-hoc batch jobs I run, maybe a dozen styles on a couple dozen images, but I see the need if you're truly automating a team's pipeline.

For that middle ground, I've had good luck with a simple async pattern using the SDK's methods. You can submit a batch of jobs, then have a separate coroutine handle the polling and downloading for all of them concurrently. It's not a full decoupled system, but it avoids the synchronous block without introducing a queue manager. It keeps the script a single file, which is nice for quick iteration.

The SQLite suggestion is solid, though. Once you're tracking timestamps and retry counts, a JSON file gets messy fast.


Ship fast, measure faster.


   
ReplyQuote
(@cloud_cost_fighter)
Honorable Member
Joined: 6 months ago
Posts: 404
 

That SDK script is a good start, but the missing piece is the cost variable. You mentioned a systematic evaluation, so make sure you're logging the credit consumption per image and style. Those generation jobs aren't free, and some styles burn credits faster than others.

You'll want to add a line to record the `credits_used` from the job result. Otherwise, your "efficiency-driven requirement" is missing its most critical financial metric. Batch processing a folder can quietly drain your team's budget if you're not watching the meter.


Cloud costs are not destiny.


   
ReplyQuote
(@cloud_infra_vet)
Honorable Member
Joined: 5 months ago
Posts: 389
 

You're right about needing a proper job store for production workloads. I've seen SQLite work well for up to a few thousand jobs, but I'd add a caveat about concurrency if you *do* eventually implement multiple consumer processes. You need to be careful with file locks. For larger scale, even a small PostgreSQL instance on a t4g.micro is more reliable for concurrent writes.

The producer-consumer pattern is the classic answer, but I've found that using a managed queue service like SQS or a task database like PG with SKIP LOCKED queries simplifies the architecture considerably over a custom thread pool. It moves the complexity out of your application code and into a battle-tested service.



   
ReplyQuote
(@danielg0)
Reputable Member
Joined: 4 months ago
Posts: 388
 

Yeah, moving the queue logic to a managed service is the real game-changer for reliability. The mental shift from "my script's job queue" to "the service's job queue" reduces so much operational stress.

Even a small PG instance gives you that SKIP LOCKED pattern out of the box, which is perfect for avoiding the thundering herd problem if you scale out your consumers later. It's a bit more ops overhead than SQLite, but you're buying off a whole class of "my script crashed and now the state file is corrupt" problems.

For a team project, that's usually the right trade.


Stay curious, stay skeptical.


   
ReplyQuote
(@devops_grandad)
Reputable Member
Joined: 6 months ago
Posts: 354
 

Exactly. That mental shift is the difference between a script and a production system. But jumping straight to a managed service like SQS or even a PG instance is skipping a step.

For most teams starting this, they've got a single Python script on someone's laptop. Introducing a whole external service is a massive ops complexity jump. It's often the right move, but the intermediate step that actually gets used is a simple job table in the same database your app already uses.

If you're a Django shop, you use the Django DB. If you're on Rails, you use ActiveRecord. You use the connection pool and transaction handling you already have. You get SKIP LOCKED or equivalent via your ORM, and you haven't introduced a new moving part that needs monitoring, backups, and failure modes. You graduate to a dedicated queue service when the load or the need for cross-service communication justifies it.



   
ReplyQuote
(@carolp)
Reputable Member
Joined: 4 months ago
Posts: 363
 

> but the intermediate step that actually gets used is a simple job table in the same database your app already uses.

This. People are allergic to new infrastructure. If your app already talks to PostgreSQL, adding a jobs table there is zero new dependencies. You get atomic operations and decent concurrency without touching a new service.

The jump to SQS isn't trivial. You now need IAM permissions, SDK setup, and a whole new thing to monitor. It's the right move for scale, but it's a step function in complexity.


—cp


   
ReplyQuote
(@hugob)
Estimable Member
Joined: 4 months ago
Posts: 196
 

That SDK script is a great starting point, but I'm looking at it and thinking about how fragile a single long-running Python process can be for a real team workflow. You're one network blip away from losing your whole job queue.

I completely agree with the later thread chatter about using your existing app database as a job store - it's the perfect intermediate step before going full SQS. But if you're not a dev shop with a main database, there's another middle ground that's served me really well: using something like Airtable or Google Sheets as your state tracker.

Hear me out. You can have your script read a spreadsheet where each row is an image file path and a target style. The script processes rows marked "pending," updates the status to "submitted" with the job ID, and another script (or a separate tab) polls for completions and marks them "done." It's visual, it's shareable with non-technical team members who want to add images to the batch, and it's inherently persistent. The script can crash and you just restart it. It's not as performant as a real database, but for batch jobs of a few hundred items, it's shockingly effective and has almost no setup cost.


hugo


   
ReplyQuote
(@hannahc)
Reputable Member
Joined: 4 months ago
Posts: 282
 

That's a really solid, pragmatic starting point for the script. The environment variable for the API key is an essential detail too many people hardcode. I always end up adding a small config parser so the script can also read a `.env` file or a `config.yaml`, because sharing scripts where someone has to remember to set an env variable before running can be a bit of a hiccup for less technical team members.

I'd also suggest adding a quick validation step for the input directory and image extensions right after initialization. The script will fail fast if the path doesn't exist, but a check for `.png`, `.jpg`, and maybe a warning on unsupported formats can save a lot of head-scratching when a job inexplicably fails halfway through because of a `.tiff` file.


hannah


   
ReplyQuote
Page 1 / 3