A Self-Hosted Observability Stack on One GCP VM

Problem

Telemetry was inconsistent and siloed. Some services had error tracking, some had a few metrics, and there was no single place to correlate them. Diagnosing a cross-service issue meant guessing and grepping across hosts.

Constraint

• Self-hosted to control cost and data residency; no secrets or internal hostnames in this writeup.
• Low overhead — collection must not meaningfully degrade request latency.
• Run lean — the whole stack had to fit comfortably on a single GCP VM.
• Every alert must map to a documented response (no alert without a runbook).

Architecture

I deployed the PLG stack plus Sentry on one GCP VM: Prometheus scrapes metrics from exporters and apps, Loki ingests logs, Grafana unifies dashboards and alerting on top of both, and Sentry captures rich, grouped application errors. Distributed tracing (OpenTelemetry → Jaeger) is the planned next layer once trace context is wired through services.

Signal responsibilities:

• Metrics — RED/USE signals and host health via Prometheus + exporters.
• Logs — centralized in Loki, queried alongside metrics in Grafana.
• Errors — Sentry for grouped, contextual exceptions with stack traces.
• Traces — OpenTelemetry → Jaeger, planned, to close the request-flow gap.

Decision / Trade-off

• PLG + Sentry on one VM over a managed SaaS. Self-hosting kept cost and data in our control; the trade-off is owning upgrades, retention, and capacity tuning ourselves.
• Grafana as the single pane. Metrics and logs share one query/alerting surface, accepting some glue work to correlate them rather than buying an all-in-one platform.
• Runbook-gated alerts. An alert ships only once its response is written down — keeps signal high and prevents fatigue, at the cost of slower alert rollout.

Result

• Metrics, logs, and errors unified into one place teams actually check.
• Cross-service issues became traceable instead of guess-and-grep.
• The full stack runs on a single GCP VM, keeping observability cost predictable.
• Every production alert ships with a documented runbook response.

What I'd watch next: wire OpenTelemetry trace context through services into Jaeger, then define SLOs per critical service and build dashboards from those SLOs — not the other way around.