I’ve been stress-testing OpenPipe’s custom node feature for the last week, specifically their “Code Node” that lets you drop in arbitrary JavaScript/TypeScript. The promise is solid: break out of their pre-built nodes to implement custom logic, data transformations, or API calls. The reality is a borderline alpha-feature level of integration that introduces more friction than flexibility.
The core issue is the execution environment. The code runs in a sandbox, which is expected, but the sandbox is severely limited and poorly documented. You don’t get a proper Node.js environment. No `fetch`? You have to use their bundled `axios` instance, but its configuration is opaque. Need a common NPM package? Tough. You’re stuck with a small, hardcoded list of pre-installed libraries. The debugging story is essentially `console.log` and praying, as error traces are truncated and often point to the wrong line.
Here’s a concrete example of a simple node meant to normalize and chunk text before sending it to an LLM. This should be trivial.
```javascript
// Attempt to split text into ~500 character chunks on sentence boundaries.
// This runs, but the lack of robust string utilities makes it clunky.
async function run(inputs) {
const text = inputs.text || '';
const sentences = text.split(/[.!?]+/);
let chunks = [];
let currentChunk = '';
for (const sentence of sentences) {
if ((currentChunk + sentence).length > 500 && currentChunk) {
chunks.push(currentChunk.trim());
currentChunk = sentence;
} else {
currentChunk += ' ' + sentence;
}
}
if (currentChunk) chunks.push(currentChunk.trim());
// Now, you'd think you could return an array directly to a downstream LLM node?
// Not so fast. Pipeline node output typing is often incompatible.
return { chunks };
}
```
The problems start after this node runs:
* The output schema is dynamic, so downstream nodes often can’t properly map `chunks` as a list input without manual JSON path wrangling.
* If an error is thrown, the entire pipeline fails with a generic "Code Node execution failed," forcing you to comment out sections and redeploy to find the culprit.
* The editor has no linting, type hints, or basic autocomplete. For anything complex, you develop locally and paste, which defeats the purpose of an integrated low-code tool.
The performance overhead is also non-trivial. In a benchmark loop of 1000 executions:
* Native OpenPipe nodes (like text splitter) processed entries in ~1.2ms avg latency.
* The equivalent custom JavaScript node averaged ~8.7ms. That’s a 7x slowdown for the same operation, which directly impacts cost-per-token when you’re billed on pipeline execution time.
My verdict after testing:
* **Use it only for** dead-simple, non-critical transformations where no native node exists.
* **Avoid for** any logic involving external API calls, complex error handling, or performance-sensitive steps.
* The feature feels like an afterthought bolted onto the pipeline designer. It’s technically “possible,” but the implementation hurdles make it a net negative for production workflows where reliability and performance are measurable requirements.
I’m curious if others have pushed this feature further and found workarounds, or if the consensus is to just build the whole logic outside of OpenPipe and use it only as a dumb inference endpoint.
Show me the benchmarks