We’ve been evaluating Langfuse for observability in a distributed event-processing system written in Python. Our architecture consists of long-running worker processes (often alive for 24+ hours) that consume messages from a queue, each message triggering a trace with several spans. We’re using the Langfuse Python SDK (`langfuse==2.0.0`) with the default configuration.
The issue we’ve encountered is a steady, monotonic increase in resident memory (RSS) of these processes over time, eventually leading to OOM kills after approximately 18 hours under production load. A heap analysis indicates the growth is tied to the Langfuse SDK’s internal buffers.
Our understanding is that the SDK batches events and flushes them asynchronously to the Langfuse backend. The memory growth suggests that either the buffer is not being cleared appropriately after successful flushes, or the flushing mechanism cannot keep pace with our event rate, causing continuous accumulation.
Here is a simplified version of our integration pattern:
```python
from langfuse import Langfuse
langfuse = Langfuse()
def process_message(message):
trace = langfuse.trace(name="message_processing")
# ... spans created within
try:
# ... business logic
trace.update(status_message="success")
except Exception as e:
trace.update(status_message="error", level="ERROR")
raise
finally:
trace.flush()
```
We’ve already verified:
* `trace.flush()` is being called in all code paths.
* The Langfuse client is instantiated once per process (as a singleton).
* Network connectivity to the Langfuse backend is stable; no persistent errors in the logs.
Key questions for the community or Langfuse team:
1. **Buffer Management**: What is the exact lifecycle of an event in the SDK’s memory? After a successful `flush()` and acknowledgement from the backend, is the event reference definitively cleared from the in-memory queue? Are there any known edge cases where garbage collection is impeded?
2. **Configuration Tuning**: Which SDK parameters directly control the flushing behavior and buffer limits? We’ve looked at `LANGFUSE_FLUSH_INTERVAL` and `LANGFUSE_FLUSH_SIZE`, but the documentation lacks details on hard memory bounds.
* Is there a maximum buffer size after which the SDK will drop events or switch to synchronous flushing?
* Should we consider implementing an explicit `langfuse.flush()` at the process level periodically, beyond the per-trace flush?
3. **Background Thread Behavior**: The SDK uses background threads for batching and sending. If the background thread encounters a transient error, does it retry indefinitely, preserving the events in memory? Is there a deadlock scenario where the producer (main thread) enqueues faster than the consumer (background thread) can process?
Our temporary mitigation has been to restart workers on a schedule, but this undermines the reliability we seek. Any insights into the SDK’s memory model or proven configurations for high-throughput, long-running processes would be greatly appreciated.
brianh