Hey folks, I've been implementing data stacks for a few years now and I keep running into the same headache. I wanted to see if anyone else is feeling this pain.
We started with a popular, fully-managed ELT vendor for our core pipelines. The promise was simple: set it and forget it. And for a while, it was great! But as we grew, we needed to pull data from a niche internal system. Their solution? Use their "SDK" to build a custom connector. Seemed reasonable, until we realized the connector only works on *their* infrastructure and can't be run anywhere else. The code we wrote is trapped there.
Now we're looking at moving to a more open pipeline tool, and we're facing a full rewrite of that logic. What's worse, their transformation layer uses a proprietary SQL dialect. So all our dbt models referencing those sources? They need a heavy edit too. The vendor's support said, "Our platform is designed for maximum reliability within our ecosystem," which really translates to, "Migrating away will be a major project."
Here's a tiny snippet of what their custom connector framework looks like. You can see it's not just Python, it's wrapped in their own classes:
```python
from vendor_sdk import BaseSource, HttpMethod
class InternalSystemSource(BaseSource):
def spec(self):
# This defines the schema for the connector config IN THEIR UI
return self.build_spec(parameters=[...])
def check(self, config):
# Health check logic - locked to their scheduler
pass
def discover(self, config):
# Catalog generation - output must match their format
pass
```
The logic itself is simple, but it's useless outside their walled garden. Has anyone else hit this? Did you bite the bullet and rewrite, or find a clever way to extract the core logic? I'm starting to think the initial speed boost wasn't worth the long-term anchor.
On the flip side, I've had better luck with tools that run on open specs (like Singer taps) or where you can at least run the connector in your own Docker container. The portability is a lifesaver.
Would love to hear your war stories and solutions.
ship it
ship it