# What Is Observability? A Practical Guide for Dev Teams URL: https://codebasechat.com/journal/what-is-observability Type: blog Locale: en Published: 2026-10-27 Updated: 2026-10-28 --- > What is observability, and how is it different from monitoring? A practical guide to logs, metrics, traces and OpenTelemetry for teams of 5 to 50 engineers. It is 2 a.m., the checkout endpoint is returning 500s for roughly one request in twenty, and the dashboards are all green. That gap, between "the system says it is fine" and "users say it is broken", is what observability exists to close. **What is observability?** It is the ability to understand what your system is doing internally by looking at the data it emits (logs, metrics and traces), well enough to answer questions you did not plan for in advance. That last clause is the whole point. Everything else in this post is detail. ## Observability vs monitoring: the question you did not write an alert for Monitoring answers questions you already knew to ask. CPU above 90%? Error rate above 2%? Queue depth growing? You pick the metrics, set thresholds, and a pager goes off when one is crossed. Observability answers the question you did not know to ask. Why do only requests from one tenant, on one app version, after a particular deploy, take four seconds longer? Nobody has an alert for that, because nobody predicted it. So the two are not rivals. Monitoring is the alerting layer. Observability is what lets you investigate once the alert fires, or once a customer writes in and no alert fired at all. If your only tool is a wall of pre-built dashboards, you have monitoring. If you can slice production data by any dimension and follow one request across services without shipping new code, you have observability. ## The three signals: metrics, logs and traces Most definitions start with "three pillars". The label is a bit stiff, but the three data types are real, and each is good at one job. - **Metrics** are numbers over time: request rate, p95 latency, memory in use. Cheap to store, fast to query, great for "is something wrong?". Poor at "which request, and why?". - **Logs** are timestamped records of discrete events: a payment failed, a retry fired, a feature flag flipped. They carry the detail, but unstructured logs on a busy service become a haystack. - **Traces** follow a single request as it crosses services. Each hop is a span with a start, a duration and some attributes. A trace is the closest thing you get to a stack trace for a distributed system. ![Printed logs next to a hand-drawn diagram of connected services](https://fdzlnqpwsaniezitwiuw.supabase.co/storage/v1/object/public/cms-media/codebasechat/2026-10/f0aa73-i1.webp) Grafana's 2025 survey of 1,255 practitioners shows where teams actually are. [Grafana's observability survey](https://grafana.com/blog/2025/03/25/observability-survey-takeaways) found that 95% of respondents use metrics and 87% use logs, but only 57% use traces. Traces are the signal most likely to help with a distributed bug, and the one fewest teams have set up. Skip the shorthand that says metrics tell you what, logs tell you why, traces tell you where. It sounds tidy and fails in practice: a good structured log often tells you more about "where" than a poorly instrumented trace. ## What makes a system observable in practice Emitting data is necessary. It is not sufficient. A system is observable when the data has three properties. **High cardinality.** You can filter by user ID, tenant, build SHA, region, feature flag. If your tool forces you to pre-aggregate by service name and status code, you cannot find the one tenant in the bad state. **Context that links signals.** A trace ID on every log line means you can jump from a slow span to the exact logs it produced. Without that link you are reading two separate stories and guessing how they connect. **Enough detail to be useful, not so much that it is unaffordable.** This is where most teams struggle. In the same Grafana survey, respondents named more than 100 different tools, services and projects between them, and the average team ran eight. Cost was a top criterion when choosing tools. Collecting everything is easy. Paying for it is the part people discover later. Here is what the starting point looks like with OpenTelemetry, the vendor-neutral instrumentation standard. This is a minimal Python example that creates a span and attaches the attributes you will want at 2 a.m.: `from opentelemetry import trace tracer = trace.get_tracer("checkout") def charge(order): with tracer.start_as_current_span("charge_card") as span: span.set_attribute("order.id", order.id) span.set_attribute("tenant.id", order.tenant_id) span.set_attribute("app.version", APP_VERSION) result = payment_client.charge(order) span.set_attribute("payment.provider", result.provider) return result`Three attributes are the difference between "checkout is slow" and "checkout is slow for tenant 4412 on version 3.8.1". The [OpenTelemetry observability primer](https://opentelemetry.io/docs/concepts/observability-primer/) explains the concepts behind this in more depth, and it is worth reading once, even if your vendor does the wiring. ## Where observability tools differ (and where they do not) Under the marketing, most platforms ingest the same three signals. They differ on pricing model, query power, and how much instrumentation they do for you. - **All-in-one platforms** bundle metrics, logs, traces, alerting and more. Convenient, with the trade-off of usage-based bills that grow with your traffic. - **Open-source stacks** such as Prometheus, Loki, Tempo and Grafana give you control and no per-host license, at the cost of running and tuning them yourself. - **OpenTelemetry-native backends** accept the standard format directly, so changing vendors does not mean re-instrumenting every service. Datadog is the reference point for the all-in-one category: broad coverage, polished UI, and a bill that deserves a monthly review. Worth it if you want one vendor and have the budget. Skip it if cost predictability is your main constraint. Grafana is the pragmatic open-source choice. You can start with dashboards on top of the data you already have, then add logs and traces as you need them. Expect to invest some engineering time. ![Distributed trace waterfall displayed on a laptop screen](https://fdzlnqpwsaniezitwiuw.supabase.co/storage/v1/object/public/cms-media/codebasechat/2026-10/fb03e7-i2.webp) SigNoz is built around OpenTelemetry from the start and puts metrics, logs and traces in one open-source interface. A reasonable pick for a team that wants the single-pane experience without the all-in-one price tag. New Relic sits between the two worlds, with a hosted platform and a free tier that is generous enough to run a real evaluation. Judge it on whether your own team can actually answer a real production question in it, not on the demo. None of these will make you observable by itself. The tool is a query interface. The instrumentation is the product. ## The part most guides skip: observability stops at the code boundary Here is a gap you will notice the first time you use any of these tools seriously. A trace tells you that `charge_card` took 3.9 seconds and that the slow part was a call into `payment_client`. It does not tell you what `payment_client` actually does, who wrote it, or why it retries three times with no backoff. For that, you leave the observability tool and go read code. In a small service that takes ten minutes. In a repo of 100K lines owned by four teams, the reading is often the longest part of the incident. You know exactly which span is slow and you still spend an hour working out how the code behind it is wired. That is the handoff problem between runtime data and source code, and it is the reason we think of observability and code understanding as two halves of one workflow: - Observability tells you **where** at runtime: which service, which span, which tenant. - Code understanding tells you **why** in the source: which function, which retry policy, which config. Teams that close the loop (span name maps to a file, a file maps to an owner and an explanation) cut the time between "alert fired" and "fix understood". It is mostly a discipline problem. Name spans after functions, put the repo path in a span attribute, and keep a short architecture note per service. An AI assistant that can answer questions about your repo, such as "where is the retry policy for payment calls set?", then has something concrete to anchor on. ![Developer working with a code editor and a metrics dashboard side by side](https://fdzlnqpwsaniezitwiuw.supabase.co/storage/v1/object/public/cms-media/codebasechat/2026-10/5fe138-i3.webp) ## Three mistakes that make teams think observability does not work **Treating dashboards as the goal.** A dashboard is a saved answer to an old question. Teams that build forty of them and never practice ad-hoc querying end up with monitoring in a nicer wrapper. Ask on every incident: did we answer this from a dashboard, or by exploring the raw data? **Sampling too early, or never.** Keeping 100% of traces on a busy service gets expensive quickly, and keeping 1% at random throws away the rare failing request you care about. Tail-based sampling, which keeps slow and failing traces and drops most of the healthy ones, is the usual compromise. Decide this before the first invoice, not after. **Letting attribute names drift.** One service writes `user_id`, another `userId`, a third `uid`. Now the query that should span all three returns a third of the picture. Agree on a small naming convention for tenant, user, version and flag attributes, and write it in the repo where people will find it. None of these are tooling problems. They are habits, and habits are cheaper to fix than vendors. ## A realistic way to start without boiling the ocean If you have nothing today, resist the urge to instrument everything in week one. A sequence that works for teams of 5 to 50 engineers: - **Pick one critical user flow.** Checkout, sign-up, the main API call. Not the whole system. - **Add structured logs with a request ID.** JSON, one event per line, trace or request ID on every line. - **Instrument that flow with OpenTelemetry traces.** Include tenant, version and flag attributes from the example above. - **Define two or three service-level objectives.** For instance, 99.5% of checkout requests succeed in under 800 ms. Alerts hang off those, not off CPU. - **Review one real incident against the data.** Could you have found the cause without new code? If not, the gap you found is your next instrumentation task. Step 5 is the one that gets skipped, and it is the only one that tells you whether you are observable. Everything else is plumbing. ## What we would actually do If your team is under 20 engineers, start with OpenTelemetry instrumentation and a backend whose pricing you can predict. The instrumentation is portable, so the backend choice is reversible. Resist buying the biggest platform because a competitor did. If you already pay for an all-in-one tool, do not rip it out. Audit what you ingest, drop the logs nobody queries, and spend the saving on instrumenting the flows that matter. And whichever way you go, write down how a span maps back to code. The dashboard shows you the fire. Somebody still has to read the source to find out why it started. ## FAQ ### What is observability in simple terms? Observability is how well you can understand what is happening inside a system from the data it produces. If you can find the cause of a problem you did not predict, using logs, metrics and traces, without deploying new code, the system is observable. ### What is the difference between observability and monitoring? Monitoring watches for known problems using predefined metrics and thresholds. Observability lets you investigate unknown problems by exploring detailed telemetry. Monitoring is the alerting layer, and observability is what you use to find the cause after an alert fires. ### What are the three pillars of observability? Metrics, logs and traces. Metrics are numbers over time, logs are records of discrete events, and traces follow a single request across services. They are most useful when linked together by shared IDs. ### Do I need OpenTelemetry to be observable? No, but it helps. OpenTelemetry is a vendor-neutral standard for emitting logs, metrics and traces. Instrumenting with it means you can change observability backends later without re-instrumenting your services. ### Is observability only for microservices? No. Distributed systems benefit the most because a single request touches many services, but a monolith with a database, a queue and a few third-party APIs also benefits from structured logs and traces. ### How much does observability cost? It depends on data volume and the tool. Hosted platforms usually charge by usage, so bills grow with traffic, while open-source stacks trade license costs for engineering time. Dropping unused logs and sampling traces are the two most common ways to control spend.