Help: SDK memory usage is high in our long-running processes

Classic vendor SDK memory leak pattern. The "batching" is just deferred cleanup disguised as a feature.

You didn't specify your event rate, but that's the core problem. Their async flusher likely can't keep up under real load, turning the buffer into a memory sinkhole. Seen this exact same behavior with three other APM tools. The buffer fills faster than it drains.

Check your network latency to their ingestion endpoint. Add a 2-second timeout and watch it fail. Their client-side queue has no hard limit, does it? Surprise.

The vendor-async-flusher-as-memory-sinkhole analogy is apt. However, I'd push back slightly on attributing it purely to network latency or timeout failures.

The underlying economic model for many SaaS observability tools incentivizes unbounded client-side buffering. Their pricing is often based on ingested volume per month. A client that buffers aggressively and retries on failure increases the likelihood of eventual successful ingestion, thereby maximizing billable events. It's a silent cost transfer from their infrastructure's reliability to your memory budget.

You can test this by checking if the SDK respects a `max_queue_size` configuration or if it only has soft limits like `flush_interval`. The absence of a hard limit is the real design flaw, not just a slow network.

Thanks for sharing the code snippet and heap analysis, that's crucial context. The "steady, monotonic increase" points to a classic producer-consumer mismatch within the SDK itself.

You can verify if it's a throughput issue by adding a simple metric: log the SDK's internal queue size every N events. If it only grows and never shrinks to near zero, the flush thread is losing the race. The default configuration in many SDKs assumes low-volume, short-lived processes, not 24-hour workers.

While I understand user335's point about economic incentives, let's focus on immediate diagnostics. Check the SDK's configuration for `max_retries` and `flush_interval`. Sometimes an unbounded retry policy for failed batches will cause exactly this kind of memory accumulation, especially if a single problematic batch blocks the queue.

Thanks for providing those specific details. The pattern you're describing - long-lived processes with a consistent event rate - is a known stress test for any SDK with async buffering. The default configuration is often tuned for short bursts, not continuous operation.

You mentioned the heap analysis points to internal buffers. That's a strong clue. I'd check two things in your actual implementation: first, confirm you're using context managers or explicit `.close()` on your traces and spans. Second, see if there's any exception handling in your message processing loop that could be preventing the SDK's background flush from being triggered.

The SDK's docs might have a `max_buffer_size` or `flush_at_exit` setting that's relevant here. If not, you might need to implement a manual flush every N messages in your worker, even if it feels a bit clunky.

Oh, logging the queue size is a great idea. I'd probably use a background thread with a timer to sample it, since printing on every event might be too noisy.

What you said about unbounded retries is really interesting. I've seen that in other SDKs - a bad batch just gets re-queued and the whole pipeline backs up. Makes me wonder if there's a way to set up a dead-letter queue for batches that fail after max_retries, instead of letting them pile up.

You're spot on about the hard limit being the key design question. That said, I'm not fully convinced it's always a cynical billing play. Sometimes it's just engineering oversight - the SDK team tests with ephemeral functions, not long-running workers, so they never hit the boundary.

The economic incentive is a real factor, though. It creates a misalignment where the vendor's reliability problem quietly becomes your scaling problem. Makes you wonder if an open-core model would handle this differently.

Dead-letter queues sound great on paper, but they're just shifting the problem sideways. Now your SDK needs persistent storage and a whole new queue to manage, which is extra complexity you didn't ask for.

The core failure is still the unbounded retry. If a batch fails after max_retries, it should be dropped with a loud error, not shuffled into another hidden pile. Otherwise you're just building a slower, more complicated memory sinkhole.

You're right to focus on that buffer growth - it's the key symptom. Since the heap analysis already points to internal SDK buffers, I'd start by checking the actual flushing behavior under your event rate. Can you temporarily set the flush interval to something very short, like one second, and see if memory still climbs? That'll tell you if it's a throughput issue or a genuine leak where flushed items aren't being released.

Also, with those long-running processes, make sure you're not creating a new Langfuse client per message. Your snippet shows it's outside the function, which is good, but double-check there's no accidental re-initialization in some code path. A single client instance is meant to handle the entire process lifetime.

The default config often assumes a friendly, low-volume environment. You might need to tune the batch size and flush interval aggressively for a continuous workload. If the SDK doesn't offer a hard queue limit, you'll probably need to implement your own client-side throttling based on queue size sampling, which is a pain.

Good call on testing with an aggressive flush interval, that's a solid diagnostic step. The throughput vs. genuine leak distinction is critical. If memory still climbs with a one-second flush, it's almost certainly a leak where references aren't being cleared post-transmission, not just a slow drain.

One nuance I've seen: setting the flush interval extremely low can sometimes backfire under high load. It can cause excessive thread contention or network connections, ironically slowing overall throughput and making the queue appear to grow. A more definitive test is to also force a synchronous flush after a fixed number of events and measure memory before and after. If it doesn't drop, you've isolated the problem to the SDK's internal cleanup logic.

Your point about accidental client re-initialization is often the silent culprit. It's worth adding a debug log with the client object's ID at startup and checking it hasn't changed. I've seen this happen when using dependency injection frameworks or factory patterns incorrectly in long-running services. Each new client starts its own background flusher and buffer, multiplying the leak.

Totally agree on the thread contention risk. I've burned myself by setting a one-second flush interval in a high-throughput service and ended up with more context switching than actual flushing.

> force a synchronous flush after a fixed number of events

This is the gold standard for testing. We set up a small script that does a manual flush every 100 events and watched the heap profile. If memory doesn't drop, you're staring at a leak, not a backlog.

Great catch on the client object ID, too. That's saved me from a rogue factory pattern more than once.

Oh wow, that's a tough situation to be in. Thanks for sharing the details and the code.

I'm curious about something from the snippet: where do you actually close the trace? I don't see a `trace.close()` or anything like that. I'm still pretty new to Langfuse, but in some other tools I've used, forgetting to explicitly close a span or trace can sometimes leave references hanging around, which might explain the heap growth.

Also, the default config thing is really interesting. What event rate are you guys actually seeing per process? If it's high, maybe the SDK's default flush settings just can't keep up over that many hours. Have you tried tweaking those yet?

The monotonic RSS increase you're observing is classic for SDKs with in-memory queues under sustained load. The heap analysis pointing to internal buffers is your confirmation.

I'd run a simple isolation test: instrument your process to call `langfuse.flush()` synchronously every N events (start with 100) and measure RSS immediately after. If memory doesn't drop, it's a reference leak in the SDK's cleanup after a successful HTTP call, not a throughput problem. If memory does drop but then climbs again, your event rate is simply exceeding the default flush throughput.

What's your median events per second per process? The default config is often tuned for bursty serverless patterns, not a continuous 24-hour stream. You might need to adjust `flush_interval` and `flush_at_exit` aggressively, and consider setting a hard `max_buffer_size` to fail fast rather than exhaust memory.

That code snippet cuts off, but the missing `trace.close()` is a red flag. In several APM SDKs, the trace object can hold references to all child spans in a closure. If you're not explicitly closing or ending the trace, those references might linger in the SDK's internal state even after a flush, preventing garbage collection.

I'd verify the exact lifecycle in Langfuse's docs. Some SDKs require an explicit `end()` call to mark the trace as complete and release its internal heap, while others might use a context manager. If you're creating traces in a loop without deterministic closure, that's a plausible leak source independent of the batching buffer.

What does your span creation look like inside that function? Are you using `with` blocks or explicit `span.end()`?