Monitoring tells you that your error rate is up. Observability tells you why. Most engineering teams have the first, believe they have the second, and discover the difference at the worst possible time: during an incident, when someone asks "which users are affected, on which versions, making which API calls" and the answer is "we don't have that data in this form."
The two terms get conflated because they live in adjacent tooling. Both involve collecting data from running systems. Both involve dashboards and alerts. But they solve different problems, and conflating them leads to investing in one while believing you have both.
Monitoring: known unknowns
Monitoring is built on metrics defined in advance. You decide what to measure: error rate, request latency at the 99th percentile, queue depth, CPU utilization. You instrument your system to record those measurements over time. A monitoring system aggregates those measurements (usually as time-series data: value X at time T) and lets you build dashboards and alert rules from them.
This works well for known failure modes. If you know that high queue depth causes problems, you alert on queue depth. If you've seen latency spikes during deployments before, you build a dashboard for deployment-time latency. Monitoring is excellent at telling you whether the failure modes you've already encountered and thought about are happening again.
The blind spot is equally clear: you can only alert on things you thought to measure. A metric you didn't define can't be in a dashboard. An alert you didn't write can't fire. Monitoring is a system for detecting known problems. The category it cannot reach is unknown failure modes, the ones you haven't seen before, haven't thought about, didn't wire up an alert for. And most interesting incidents involve novel failure modes. The well-understood ones tend to get fixed or mitigated; what remains is the unexpected.
Observability: unknown unknowns
Observability (in the engineering sense, borrowed from control theory) is the ability to infer the internal state of a system from its external outputs. In practice: can you answer arbitrary questions about what your system is doing, without modifying or redeploying it?
The "arbitrary questions" part is what distinguishes it from monitoring. During an incident, you don't know in advance what question you'll need to ask. You discover that something is wrong and you need to find out why. "Is this affecting all users or just users on iOS?" "Is it every request or only requests that hit a particular database shard?" "Did it start when we deployed version 1.4.7?" These questions can't be answered by a dashboard you built last month, because you didn't know to build it for this specific incident.
Observability requires data that is high-cardinality (can distinguish individual requests, individual users, individual hosts) and high-dimensionality (many attributes per event). With that data, you can investigate any incident by querying what you already have. You're not limited to the questions you predefined; you're exploring the data to find the pattern that explains the behavior.
These are structurally different models. Monitoring is a pull model: you define the questions in advance, the system answers them continuously, and you check the answers when something goes wrong. Observability is an exploratory model: when something goes wrong, you ask questions against a rich dataset and follow the evidence wherever it leads.
The three pillars
Metrics, logs, and traces are the standard data types in modern observability. Each has a distinct structure, distinct storage characteristics, and distinct use cases. Getting the most from observability means understanding what each is good at.
Metrics are aggregated numerical measurements over time. Your monitoring system stores
http_requests_total{status="500"} = 42 at each timestamp. They are cheap
to store and query because they are pre-aggregated. They are excellent for dashboards
(you can see the trend over days or months without storing every individual request)
and for alerting (you evaluate a threshold against a number, which is fast). The
limitation is that aggregation discards detail. Once you have counted "42 errors in
the last minute," you cannot go back and ask which requests those were.
Logs are structured records of individual events. A log entry captures everything about a specific request, operation, or system event at the moment it occurred. They are expensive at scale (a high-traffic service can generate millions of log events per second) but they preserve full detail. Logs are the right tool for debugging a specific event: "tell me everything about the request with ID abc123." The cost and verbosity make them impractical as the primary data source for aggregate analysis.
Traces track the causal chain of a request through a distributed system. A single user request might pass through a load balancer, an API gateway, three microservices, and a database. A trace records each hop, how long it took, and what happened, stitched together by a common trace ID that propagates through every service's calls. Traces answer "which service is slow and what is it calling" in a way that logs from individual services cannot, because you can follow a single request across service boundaries.
The tools for each: Prometheus for metrics, the ELK stack (Elasticsearch, Logstash, Kibana) or Loki for logs, Jaeger or Zipkin (or cloud-native equivalents) for traces. OpenTelemetry, a set of vendor-neutral APIs and SDKs, is the emerging standard for instrumenting services to produce all three. You instrument once, and the data flows to whichever backends you choose.
Structured logging vs. log strings
The difference between useful logs and useless logs is structure. Consider two ways to log the same event:
"ERROR: request abc for user 42 failed after 1240ms: timeout connecting to payment service"{"level":"error","request_id":"abc","user_id":42,"duration_ms":1240,"error":"timeout","downstream":"payment-service","timestamp":"2026-05-31T14:23:01Z"}The first is a string. To search it, you use grep (or an equivalent). You can find "all errors mentioning the payment service," but you can't aggregate duration across errors, filter by user ID without parsing strings, or join these log lines to other data. At low volume, it's workable. At scale, grep is a nightmare: slow, brittle to format changes, and incapable of aggregation.
The second is structured. Each field is a typed key-value pair. You can query
level="error" AND downstream="payment-service", group by user_id,
compute the 95th percentile of duration_ms, filter to the last 15 minutes. You can
join on request_id to correlate with traces. You can build dashboards from these
without writing any special parsing logic. Structured logs are the prerequisite for
using log data analytically rather than just textually.
The discipline of structured logging is straightforward: never log bare strings with interpolated values; always log a record with typed fields. Every field that you might want to filter or group by should be a first-class key, not embedded in a message string.
The high-cardinality problem
There's a structural reason why traditional monitoring tools can't give you per-request observability, and it's worth understanding because it explains why purpose-built observability backends exist.
Prometheus (and most metrics systems) stores time series. A time series is a metric
name plus a set of labels. http_requests_total{method="POST",status="500"}
is one time series. The number of time series is the product of all label cardinalities.
If you have 2 methods, 4 status codes, and 10 endpoints, that's 80 time series,
perfectly manageable.
Now add user_id as a label. With a million users, you have 80 million
time series. Prometheus cannot store that. The storage model (pre-aggregated values
at fixed time intervals) is incompatible with high-cardinality labels. This is a
fundamental property of the data model, not a performance limitation you can tune
away.
True observability backends store individual events and query them at read time. When you ask "what's the latency for requests from user 42," the backend scans the raw events and computes the answer. This is more expensive at query time than a pre-aggregated metric, but it enables arbitrary queries over high-cardinality dimensions. Honeycomb is the canonical example; Grafana Tempo (for traces) and ClickHouse-based log backends take similar approaches. The tradeoff is cost (raw events are expensive to store) and query latency, in exchange for the ability to answer questions you didn't predefine.
The answer to "why"
Observability isn't three dashboards. It's not a product category or a list of tools. It's the property of a system where you can ask "what is this specific request doing and why is it slow" and get a meaningful answer, without knowing in advance that you'd need to ask.
Monitoring handles known failure modes efficiently. It should run in every production system; it catches the expected problems early. Observability handles the failure modes you haven't predicted. It's the difference between being able to detect that something is wrong and being able to understand what and why with enough precision to fix it.
The gap between the two matters most during incidents, precisely the moments when you least want to discover that the data you need doesn't exist. Investing in structured logging, distributed tracing, and high-cardinality event storage feels like overhead until the first incident where you can answer "which users, which requests, which version, which service" in five minutes instead of two hours. After that, it feels like infrastructure.
Back to writing