What makes up an observability stack?

Collection agents or a collector, storage and query for metrics, logs, and traces, an alerting component, and visualisation. The pillars have different cost profiles, so mixed stacks combining self-managed and hosted components for different data types are common and reasonable rather than a compromise.

Why do observability costs grow unexpectedly?

Usually cardinality. A metric's cost depends on its distinct label combinations, so adding a label with unbounded values such as a customer or request identifier multiplies the series count enormously. Keep unbounded values in logs or trace attributes, where the storage model expects them.

Should you self-manage or use a hosted platform?

It depends on data volume, available engineering time, and how predictable the cost must be. Self-managed gives control and predictable infrastructure cost while you own the reliability of the system that tells you other systems are broken. Hosted removes that burden and couples your bill to casual engineering decisions.

What should you instrument first?

Symptom-based alerts on what users experience: error rate, high-percentile latency, and basic availability. They catch most real incidents with minimal infrastructure. Alerting on causes instead generates notifications for conditions nobody noticed, which trains people to ignore the alerts that matter.

How do you avoid vendor lock-in with observability?

Instrument with a vendor-neutral standard and put a collector between your applications and your backend. Changing backend then becomes a collector configuration change rather than reinstrumenting every service, which matters because whatever you choose will be reconsidered within a couple of years.

How to Assemble an Observability Stack

An observability stack collects metrics, logs, and traces, then stores, queries, and alerts on them. Component choice matters less than two decisions: how you control cardinality and retention, and whether instrumentation is vendor neutral, because those determine both cost and your ability to change tools later.

The components and what each is for

Metrics. Numeric time series, cheap to store and query, suited to alerting and to answering how much and how often. Their cost scales with the number of distinct label combinations rather than with request volume, which is the single most important property to understand before choosing anything.

Logs. Discrete events with detail, suited to answering what exactly happened in this case. The most expensive pillar, because volume grows with traffic and useful retention periods are long.

Traces. Causally linked spans across services, suited to answering where the time went in a request that crossed six boundaries. Indispensable in distributed systems and usually sampled, since keeping every trace is rarely affordable or necessary.

Collection. An agent or collector receiving telemetry, batching, enriching, and forwarding it. Putting a vendor-neutral collector between your applications and your backend is what makes changing the backend a configuration change rather than a reinstrumentation project.

Storage and query. Where data lands and how it is searched, which is where most of the cost and most of the operational burden sits.

Alerting. Evaluating conditions and notifying someone. Worth treating as a first-class component rather than a feature of a dashboard tool, since it is the part that wakes people up.

Dashboards. Visualisation. Useful and consistently overvalued relative to alerting and querying.

Why cardinality is the cost driver

A metric's cost is a function of its distinct label combinations. A request counter labelled by endpoint and status code is a handful of series. Add customer identifier and the count multiplies by your customer base; add request identifier and you have created a series per request, which is a log entry wearing a metric's clothing. This is how observability bills grow by an order of magnitude after an innocuous label is added, and it is a design property rather than a pricing surprise. The discipline is to keep unbounded values out of metric labels and put them in logs or trace attributes, where the storage model expects them.

Choosing between hosted and self-managed

Hosted platforms remove the operational burden of running storage at scale, which is substantial and unglamorous work. You pay per volume ingested and retained, which means your bill is directly coupled to decisions engineers make casually, such as adding a label or raising a log level. Cost controls therefore have to be organisational as well as technical.

Self-managed open source gives predictable infrastructure cost and full control over retention, in exchange for owning the reliability of a system whose primary purpose is telling you when other systems are unreliable. That inversion matters: the observability stack failing during an incident is a specific and common failure mode, and it needs to be in a different failure domain from what it watches.

The honest comparison is total cost. Self-managed is not cheaper if it consumes an engineer's time permanently, and hosted is not cheaper if the bill grows without bound. The variables are your data volume, how much engineering time you have, and how predictable you need the cost to be.

Mixed stacks are normal and defensible. Self-managed metrics with hosted traces, or hosted metrics with logs in cheap object storage, are common because the pillars have genuinely different cost profiles.

Migration cost is the thing to minimise. Whatever you choose will be reconsidered in two years. Vendor-neutral instrumentation and a collector in the middle mean that reconsideration is a project of weeks rather than a rewrite, and that is worth more than any feature comparison.

Build order that works

Symptom-based alerting first. A handful of alerts on things users experience: error rate, latency at a high percentile, and whether the service responds at all. These catch most real incidents and need very little infrastructure. Alerting on causes rather than symptoms produces a stream of notifications for conditions users never noticed.

Then structured logs with identifiers. Structured from the beginning, because converting a corpus of unstructured logs later is miserable. Include a request identifier propagated across services; this is the cheapest thing you can do to make debugging tractable and the most annoying to retrofit.

Then traces on the paths that matter. Start with the request flows that are slow or complicated rather than instrumenting everything. Sample aggressively, and keep traces that contain errors.

Then dashboards, driven by real questions. Build the view that answers a question you actually had during an incident. Dashboards built speculatively are not read, and their existence implies coverage that is not there.

Then cost controls, before you need them. Retention tiers, log sampling for high-volume low-value events, and a cardinality budget per service. Retrofitting these after a bill shock means deleting data people have started relying on.

Correlate with consistent identifiers. The value of the three pillars together comes from moving between them: an alert to the traces in that window to the logs for one failing request. That only works if the same identifiers appear in all three, decided early and enforced in a shared library.

Keep the stack out of the blast radius. If it shares infrastructure, identity, or network path with what it monitors, it will be unavailable exactly when needed.

Frequently asked questions

What makes up an observability stack?: Collection agents or a collector, storage and query for metrics, logs, and traces, an alerting component, and visualisation. The pillars have different cost profiles, so mixed stacks combining self-managed and hosted components for different data types are common and reasonable rather than a compromise.
Why do observability costs grow unexpectedly?: Usually cardinality. A metric's cost depends on its distinct label combinations, so adding a label with unbounded values such as a customer or request identifier multiplies the series count enormously. Keep unbounded values in logs or trace attributes, where the storage model expects them.
Should you self-manage or use a hosted platform?: It depends on data volume, available engineering time, and how predictable the cost must be. Self-managed gives control and predictable infrastructure cost while you own the reliability of the system that tells you other systems are broken. Hosted removes that burden and couples your bill to casual engineering decisions.
What should you instrument first?: Symptom-based alerts on what users experience: error rate, high-percentile latency, and basic availability. They catch most real incidents with minimal infrastructure. Alerting on causes instead generates notifications for conditions nobody noticed, which trains people to ignore the alerts that matter.
How do you avoid vendor lock-in with observability?: Instrument with a vendor-neutral standard and put a collector between your applications and your backend. Changing backend then becomes a collector configuration change rather than reinstrumenting every service, which matters because whatever you choose will be reconsidered within a couple of years.