Alright, let's stir the pot a bit. I keep seeing teams, especially those running a classic monolithic application, immediately reach for the big, expensive, all-in-one observability platforms. They get dazzled by distributed tracing and infinite cardinality. But I've migrated enough of these systems (and off of them) to say: **you're probably overcomplicating it and burning cash.**
For a single, well-understood monolith, the core needs are simple:
1. **Log aggregation** with decent search.
2. **System metrics** (CPU, memory, disk I/O).
3. **Application metrics** (request rate, error rate, latency).
4. **Alerting** that reliably wakes you up.
You can achieve a rock-solid, understandable, and *cheap* observability stack with OSS or built-in cloud services. The moment you introduce a fancy SaaS tool, you're paying a premium for features you don't use and dealing with its opinionated data model. I've seen teams spend more time wrestling with an APM agent's overhead and sampling than actually debugging their app.
Here's a battle-tested, rollback-friendly stack I've implemented more than once:
* **Logs:** `vector` or `fluentd` scraping files -> `loki` (for the monolithic log use case, it's simpler and cheaper than the Elastic stack). The query language is LogQL, which is plenty powerful.
* **Metrics:** Good ol' `node_exporter`/`windows_exporter` and your app's Prometheus client -> `prometheus` for local storage. Long-term? `victoriametrics` is a fantastic, less finicky alternative for long-term storage.
* **Alerting:** `alertmanager`. It just works. Define your alerts in PromQL.
* **Dashboarding:** `grafana` (it ties logs from Loki and metrics from Prometheus together beautifully).
The configs are straightforward. A Prometheus scrape config for your monolith might look like:
```yaml
scrape_configs:
- job_name: 'my_monolith'
static_configs:
- targets: ['app-host:8080']
metrics_path: '/actuator/prometheus'
```
The **gotchas** with the fancy tools in a monolith context?
* **Cost Explosion:** They often charge per data point, per span, per GB ingested. A chatty monolith can generate a surprising amount of "spans" that are just nested functions, costing you dearly.
* **Vendor Lock-in:** Their proprietary query languages and agents make migration a multi-month project (trust me, I'm the `migration_warrior`).
* **Cardinality Anxiety:** You start worrying about labels instead of just instrumenting your code.
Now, I'm not saying never upgrade. When you start splitting that monolith, *that's* when you re-evaluate. But start simple. Master the fundamentals with tools you control. You'll understand your system better, and your CFO will thank you.
So, challenge for the group: **What's in your "just enough" observability stack for a monolith?** Anyone else done a migration *off* of a heavyweight platform back to something leaner? What was the breaking point?
Backup twice, migrate once.
Totally agree on the core needs list. Where I've seen teams get tripped up is underestimating the "decent search" part for logs. Loki's great and cheap, but its query language can be a speed bump for devs used to grepping files or basic Kibana. You almost need to treat setting up a few key label matchers and common queries as part of the app's documentation.
Also, for that application metrics piece, a dead-simple Prometheus exporter built into the monolith gives you so much control for nearly free. It forces you to think about what you actually care to measure. The moment you plug in an APM agent, you get a thousand default metrics and the signal gets buried.