Hello everyone,
I’ve been reading through the forum for a few weeks now, trying to absorb as much as I can before posting. There’s a lot of great information here, and I appreciate the detailed discussions I’ve seen so far. I’m relatively new to OpenPipe, but I come from a background working with ERP systems like NetSuite, where scheduling and automation are critical for inventory and supply chain workflows.
I’m currently setting up a pipeline to generate daily production reports by pulling data from several integrated sources, transforming it, and then pushing a summary to a dashboard. The pipeline itself is working correctly, but I’ve hit a configuration question I couldn’t fully resolve by reviewing the documentation and existing threads.
My requirement is that this pipeline should only execute on weekdays (Monday through Friday). The data sources we rely on, primarily from our manufacturing and logistics partners, are not updated on weekends, so running the pipeline on Saturday or Sunday would be redundant and would consume resources unnecessarily. In my past work with other orchestration tools, I’ve used cron expressions or specific scheduler UI settings to handle this kind of restriction.
In OpenPipe, I see the option to set a schedule with a cron-like pattern in the pipeline settings. I’ve experimented with something like `0 8 * * 1-5` to run at 8 AM from Monday to Friday. However, I am cautious about getting this exactly right, as a mistake could lead to missed reports on a Monday or incorrect runs over the weekend. I also noticed there is a “Skip on Failures” setting, but that seems related to error handling, not day-based exclusions.
Could someone with more experience confirm the correct way to define a weekday-only schedule within OpenPipe? Specifically, I’d like to know:
- Is the standard cron expression (`1-5` for weekdays) the officially supported and reliable method?
- Are there any timezone considerations I should be aware of? The pipeline uses UTC in its logs, but our team and data sources operate in Eastern Time.
- Has anyone encountered issues with holiday schedules? For now, running on standard business holidays is acceptable, but I’m curious if there’s a built-in way to handle annual date exclusions, or if that requires a different approach, like a conditional check at the start of the pipeline.
I want to ensure my configuration is robust from the start, especially since this will feed into other systems. Thank you in advance for any guidance you can provide.
Right, the weekday scheduling question is a common one when moving from a traditional cron-based system. Your approach with cron expressions is still valid in OpenPipe's scheduler.
You'll need to configure the pipeline's schedule using a cron pattern. Something like `0 8 * * 1-5` would run it at 8 AM Monday through Friday. The key detail people often miss is ensuring the timezone is correctly set on the schedule object, otherwise you might get unexpected runs based on UTC.
A caveat from a cost perspective: even with correct scheduling, verify your compute configuration for the pipeline doesn't have a minimum node count that sits idle over the weekend. Sometimes the scheduler stops the execution, but the underlying resources aren't fully scaled down, which still incurs cost.
Every dollar counts.