lgtm-sidecar: a resource-capped observability stack for small teams

Challenge

Small teams either pay a SaaS observability bill that scales with their success, or install Grafana on the app server and watch it eat the memory the app needed. Neither is acceptable for a company running a dozen services on a modest AWS footprint. I wanted all four signals, correlated, on hardware we own, without the observability stack ever being the cause of an incident.

Approach

Decouple it

The stack runs on its own EC2 host. Applications ship telemetry through a lightweight Edge Alloy agent running next to them; the heavy components (Loki, Tempo, Prometheus, Pyroscope, Grafana, Alertmanager) live on the sidecar host behind Nginx + Certbot TLS. If the sidecar dies, applications keep running and buffer briefly.

Cap everything

Every container has memory limits. Alloy pipelines are memory-limited. Traces are tail-sampled with a rate cap: errors and slow requests are always kept, happy-path traffic is sampled down to a budget. Loki and Tempo write to S3 via an IAM role, so retention is a bucket lifecycle policy rather than a disk alarm.

Correlate by design

OpenTelemetry across FastAPI and Next.js with W3C TraceContext propagation, so a click in the browser and the database query it caused share a trace id. Tenant and office identifiers travel in span baggage and land in structured log metadata, never as high-cardinality index labels. The rationale is the same one I apply to multi-tenant data in Designing Multi-Tenant Database Architectures: isolate by design, not by discipline.

Alert like you mean it

Prometheus rules and Loki ruler feed Alertmanager, which routes to Slack and email with grouping and inhibition so one outage produces one page. Dashboards ship as code with the repo.

Architecture

What shipped

• A docker compose deployment with sane limits, TLS and dashboards.
• Edge agent config for any host: one file, one environment variable.
• Tracing instrumentation guides for FastAPI and Next.js.
• Alert routing templates for Slack and email.

Results

• Root-cause analysis for cross-service incidents went from "grep three servers" to "open the trace".
• The observability host has never caused an application incident; its own resource usage is flat and predictable.
• The same stack now watches the systems in the infra-ops: one Terraform + Ansible platform for dev, staging and prod and CaseCloud: multi-tenant legal SaaS with RAG over 80M+ judgments write-ups.

What I'd do differently

• Add SLO objects (burn-rate alerts) as the default alerting primitive rather than threshold rules; they page less and mean more.
• Ship a Helm chart alongside the compose file, since the next deployment target is Kubernetes.