That's a great breakdown, and thanks for sharing it. The observability pipeline point really resonates. We spent a lot of time getting logs into Datadog, and the custom parsing required was more than we expected. It felt like we were building a connector from scratch rather than using an off-the-shelf integration.
For the API, we found it was comprehensive for basic CRUD operations, but it lacked the idempotency we needed for our automation. We had to build that safety check layer ourselves, which added complexity. Did your team run into similar issues with policy updates, or was the integration smoother on that front?
Totally agree about that extra plumbing. We're just setting up our ELK stack now and you're making me nervous about integrating with it. 😅
Did you end up using their default log forwarder or go with a different agent from the start? I'm trying to decide which path is more work in the long run.
We didn't, and it was a problem. Our translator service crashed silently for two days last month. We only caught it when someone noticed a new service wasn't showing up in the policy console.
We're adding a simple health check endpoint now that pings the policy API to verify the last update timestamp. It's not perfect, but at least we'll get an alert. What kind of heartbeat did you have in mind?