Sample 03 • Tool Selection • 14 min
How to Choose a Cloud-Native Observability Stack
Observability is the one place where “buy vs build” goes wrong most often. The right answer depends less on features and more on MTTR vs. log volume vs. retention vs. team size. This guide gives you an OTel-first spine and cost-aware breakpoints so you pick intentionally.
Start with one rule: OTel first, vendor second
Instrument once with OpenTelemetry (traces, metrics, logs) and export to whatever backend wins today. Vendor agents that lock you into proprietary instrumentation are a future migration tax. Even if you buy Datadog/Honeycomb, front it with OTel Collectors — you’ll thank yourself at renewal.
- Collector pattern: app → OTel SDK → OTel Collector (sidecar or gateway) → backend(s). Lets you sample, redact PII, and dual-write during a switch.
- Don’t roll your own SDKs. Use OTel auto-instrumentation where available (Java, Node, Python, Go) and hand-instrument only hot paths.
The four-way tradeoff you actually need
Forget “best observability tool” lists. Score each on four axes for your workload:
| Axis | Favors OSS (Grafana + Loki/Mimir/Tempo/Pyroscope / ELK) | Favors Commercial (Datadog, New Relic, Honeycomb) | Breakpoint |
|---|---|---|---|
| MTTR criticality | Team can live with assembled views | MTTR is revenue; needs correlated APM out-of-box | If >30 min MTTR = revenue loss → lean commercial APM |
| Log volume | <200 GB/day, >30d retention wanted → Loki/Mimir on S3 is 40–60% cheaper | <50 GB/day, <7d retention; simplicity wins → vendor is fine | ~100 GB/day is the cost inflection |
| Team size | ≥2 SREs to own retention, cardinality, upgrades | ≤6 engineers total → paying for “it just works” beats staffing | No SRE? Pay for ops. |
| Trace needs | Moderate sampling, backend in S3 (Tempo) is cheap | Need tail-based sampling + columnar queries (Honeycomb) | High-cardinality debugging → Honeycomb shines |
Cost breakpoints (illustrative, verify your volume)
Numbers assume ~50 hosts, ~100 GB/day logs, ~10k spans/sec. Get vendor quotes with your actual cardinality — cardinality is where bills explode.
- Grafana Cloud (Loki + Mimir + Tempo) + S3: often 40–60% lower than per-host APM at scale, but you own dashboards, alerts, retention math, and Collector tuning.
- Datadog / New Relic: higher baseline ($15–$35/host + ingest), but you get APM + RUM + SLO + security in one pane — and a lot less pager design work.
- Honeycomb: best-in-class for wide events + high-cardinality trace queries; strongest for services-heavy debugging, weaker as a budget log warehouse.
Hidden cost warning: log cardinality (labels) and high-tag metrics are the #1 bill shock. Enforce label budgets and sampling at the Collector — before data hits the backend.
Use X instead of Y — when… (copy to your ADR)
| Use | Instead of | When |
|---|---|---|
| Grafana + Loki + Mimir + Tempo (or Grafana Cloud) | ELK (self-hosted) | You want log cost control and already speak PromQL; ELK is heavier to run well |
| Datadog / New Relic | DIY Prometheus + Grafana + ELK assembly | MTTR is urgent and team ≤10; time-to-insight > tooling cost |
| Honeycomb | Jaeger/Tempo for tracing only | You debug via traces + high-cardinality fields (tenant, build, feature flag) daily |
| OTel Collector Gateway | Vendor agent per host | Always — unless you deliberately accept lock-in for ≤12 mo |
| Pyroscope / Parca | No continuous profiling | When you chase latency tails or alloc pressure beyond traces |
What we’d pick for common teams
- 5-engineer SaaS, ≤50 GB/day, <7d log retention → OTel → Datadog or Grafana Cloud (whichever is cheaper at your vendor rep quote). Optimize for MTTR.
- Platform team of 2–3, ≥100 GB/day, 30d+ retention → OTel → Grafana stack (self-host Collectors + Grafana Cloud or self-host Mimir/Loki on S3). You will save meaningfully.
- Services-heavy with hard debug → OTel → Honeycomb for tracing + Grafana/Loki for logs/metrics. Best of both; tolerate two UIs.
Implementation guardrails
- Cardinality budget: cap label values (e.g., no tenant ID as a raw Prometheus label). Aggregate high-card at the app.
- Sampling: head-based at first; graduate to tail-based (Collector) when trace volume >50k/s or P95 matters.
- SLOs, not alerts everywhere: start with 4 golden signals + SLO burn alerts; silence the rest.
- Dual-write for 2 weeks on any switch: prove parity before cutting over.
Explicit recommendation
Default: OTel Collector Gateway + Grafana Cloud (or self-host Mimir/Loki/Tempo on S3 if you have SREs) for logs/metrics/traces. Swap to Datadog/New Relic when you have no SRE and every outage minute is expensive. Add Honeycomb when tracing is your daily debugging surface and high-cardinality queries are non-negotiable. In all cases: enforce label budgets and sampling at the Collector.