I've always been fascinated by the concept of a fully in-memory, event-driven content pipeline. The typical workflow of generating a file, saving it, having another process pick it up, edit it, and then push it, introduces latency and operational overhead. I've just completed a prototype that eliminates the filesystem from the equation entirely, using a composition of cloud-native services and a state machine to manage the entire lifecycle.
The core architecture leverages Google Cloud Platform services, orchestrated with Cloud Workflows and Pub/Sub. The pipeline is triggered by a simple HTTP request containing a topic and a basic brief. This event publishes a message to a Pub/Sub topic, which initiates the workflow. The state machine then coordinates the following stages sequentially, passing the payload as a JSON object between services:
* **Generation:** The initial Pub/Sub message triggers a Cloud Function (or a Cloud Run service) that calls the OpenAI API. The function uses a structured prompt template, injected with the brief from the triggering event, to generate a draft article. The output (title, body, metadata) is added to the evolving payload.
* **Review & Human-in-the-Loop:** The enriched payload is then sent to a dedicated Cloud Run service that exposes a simple web interface. This service writes the draft to a temporary Firestore collection, generating a unique review URL. This URL is dispatched via email to an editor using SendGrid's API. The workflow pauses here, using an event arc in Cloud Workflows, waiting for a `review.approved` or `review.revisions` callback to the workflow's webhook endpoint. The editor makes changes directly in the web UI, which submits back to the webhook, resuming the workflow with the edited content.
* **Publishing:** The final, approved payload is delivered to another Cloud Function configured as the webhook target for our headless CMS (Sanity.io). The function maps the payload fields to the CMS's schema and performs a direct mutation via its API to create and publish the document. A final success message is logged to Cloud Logging and a summary is posted to a Slack channel.
The key to the "no file" paradigm is the persistent, serialized payload within the workflow execution and the use of Firestore solely as a temporary, structured data store for the human edit interface—not as a file repository. Here is a simplified excerpt of the Cloud Workflows definition, showcasing the state structure:
```yaml
main:
params: [event]
steps:
- generate_draft:
call: http.post
args:
url: ${GEN_FUNCTION_URL}
body:
brief: ${event.brief}
topic: ${event.topic}
result: draft_result
- create_review_task:
call: http.post
args:
url: ${REVIEW_SERVICE_URL}/create
body: ${draft_result.body}
result: review_task
- await_human_review:
call: events.await
args:
event: "review_event"
callback:
http_callback:
url: ${WORKFLOW_WEBHOOK_URL}
result: reviewed_content
- publish_to_cms:
call: http.post
args:
url: ${CMS_PUBLISH_FUNCTION_URL}
body: ${reviewed_content.body}
result: publish_result
- return_result:
return: ${publish_result}
```
This approach has several notable advantages. Latency between stages is reduced to network call times, and we gain complete observability through a single Cloud Workflows execution trace. The pipeline is also inherently scalable and portable; swapping the LLM, the review interface, or the CMS destination requires modifying only one isolated service. The primary consideration is ensuring payload size remains within service limits (Cloud Workflows has a 1MB execution log limit), but for structured article content, this is seldom an issue. I'm now experimenting with adding a parallel dbt Cloud webhook step after publishing to kick off downstream analytics on publication metadata.
Extract, transform, trust
Cool party trick. But the cost model on this, especially with OpenAI API calls in a state machine, is a total black box until you get the bill. How do you track or cap the spend when everything's a chained event with no hard stops?
—EB