Hey folks! 👋 Been using ElevenLabs for a few months now to generate voiceovers for our internal tooling tutorials and notifications. The quality is fantastic, but I've hit a scaling question I figured this community could help with.
Our pipeline currently calls the API, gets the MP3, and stores it directly in our app's repo under `/assets`. This was fine for a handful of files, but now we're generating hundreds per week. It's blowing up our repo size, and versioning binary files in Git feels wrong. Build times are creeping up because of the clone/download size.
What's the best practice for handling these generated audio files at scale? I'm thinking along the lines of:
- **Object Storage:** Something like S3, GCS, or Azure Blob Storage seems obvious. Do you set lifecycle rules to archive old files?
- **Metadata Management:** How do you track which audio file corresponds to which script/version/voice? A simple metadata DB or just rely on structured object keys?
- **Pipeline Integration:** For those using CI/CD (GitHub Actions/GitLab CI), how do you orchestrate the generation *and* storage? Do you have a step that uploads the artifact directly to storage and then maybe stores just the public URL in your app config?
Here's a simplified snippet of what our current GitHub Actions step looks like:
```yaml
- name: Generate and Store Audio
run: |
RESPONSE=$(curl -X POST "https://api.elevenlabs.io/v1/text-to-speech/$VOICE_ID"
-H "xi-api-key: ${{ secrets.ELEVENLABS_API_KEY }}"
-H "Content-Type: application/json"
-d "{"text": "${{ steps.generate_text.outputs.script }}"}"
--output audio.mp3)
# Currently just commits it to the repo
git add assets/audio.mp3
git commit -m "Add generated audio"
```
This feels brittle. Would love to hear how others are structuring this, especially if you're automating it heavily. What's your object naming convention? Do you handle cleanup? Any pitfalls with CDN caching for frequently updated audio?
-pipelinepilot
Pipeline Pilot
I'm a senior DevOps engineer at a mid-sized EdTech company. We generate a massive amount of audio and video content daily, and I manage the pipeline that stores and serves it all. We run on AWS, use Terraform and Ansible, and serve files to around half a million users.
Here's a breakdown of what actually matters, based on burning through a few bad designs:
1. **Cost at Scale:** Object storage like S3 is dirt cheap for storage ($0.023/GB for Standard) but the killer is egress and requests. If your internal app hits these files constantly, you'll pay. We put CloudFront in front and saved about 40% on the bill. Lifecycle rules are non-negotiable; we transition to Infrequent Access after 30 days and delete after 365. Your "hundreds per week" will become tens of thousands, and you don't want to pay Standard rates for old tutorial clips.
2. **Metadata Linkage:** Do NOT try to manage this in filenames or a separate metadata database. That's a sync nightmare. Every audio file we generate gets a deterministic UUID based on the script content and voice parameters. We store that as the object key (e.g., `{uuid}.mp3`). The *only* source of truth is a Postgres table that maps the business context (e.g., `tutorial_video_id`, `voice_id`, `script_hash`) to that UUID/filename. The app looks up the UUID and requests the file from the CDN. This makes caching trivial and lookups fast.
3. **Pipeline Integration:** Your CI/CD should never commit binaries to git. Full stop. Our generation is a GitHub Actions job that calls the voice API, streams the MP3 directly to an S3 bucket (using the AWS CLI), and on success, writes the metadata to our application database. The commit only contains the script text and generation parameters. The entire artifact is the object in S3. This keeps repo size flat and build times predictable.
4. **Performance & Limits:** At our peak, we handle about 1,200 generation requests per hour. The bottleneck is never S3; it's the API quota and cost from ElevenLabs or similar. We implemented a local, simple cache layer: before any API call, we compute the hash of the script and voice settings. If that hash exists in our metadata table, we skip generation and serve the existing S3 object. This cut our external API calls by roughly 70%. S3 can handle the request load, but you must design for eventual consistency on write if you have parallel jobs generating the same file.
My pick is **AWS S3 with CloudFront and a lifecycle policy**, paired with a **Postgres metadata table**. This works for any cloud, really - just swap in GCS or Azure Blobs. If you're not on AWS, the equivalent object storage service is fine. The crucial part is the decoupled metadata database and deterministic hashing to avoid duplicate generations. Tell me your cloud provider and whether your app is cloud-hosted or on-prem, and I'll give you the exact Terraform snippet for the bucket policy.