Monitoring, logging and tracing

Think of it like this

Analogy

A car's dashboard shows a few key numbers at a glance: speed, fuel, engine temperature. Those are metrics. The mechanic's service history lists every event in detail: “replaced brake pads, March 3”. Those are logs. A GPS track of one journey shows exactly where you were slow and where you stopped. That's a trace. You need all three to understand why the car is acting up.

The problem it solves

With one server you could log in and look around. With dozens of services and hundreds of instances, a single user request might touch ten machines. When something breaks, you need to answer quickly: is it broken, for whom, since when, where, and why? Without good observability, outages last longer, root causes stay unknown, and the same problems come back.

A real example

Example

Checkout slows down at 14:05

At 14:05 an alert fires: checkout's p99 latency has broken its target for 5 minutes. The on-call engineer opens the dashboard. Metrics show the payment service's latency jumping at 14:00.

They open a slow trace from that period. The trace shows the payment service waiting 2 seconds on a call to the fraud service, and inside that, a single database query taking almost all of the time.

They search the logs for that trace's request ID. The fraud service's logs show a migration at 13:58 that dropped an index. They add it back, and latency returns to normal by 14:20. Metrics showed something was wrong, the trace showed where, and the logs showed why.

How it works

The three signals

Signal	What it is	Answers	Example
Metrics	Numbers over time, cheap to store	Is something wrong? How bad?	Requests/s, error rate, p99 latency, queue depth
Logs	Timestamped records of events	What exactly happened?	{"level":"error","msg":"card declined","order":42}
Traces	One request's path across services, as timed spans	Where did the time go?	API 820 ms → payments 790 ms → card network 770 ms

What to measure

Two well-known checklists:

• The four golden signals (for services): latency, traffic, errors, saturation.
• RED (for request-driven services): Rate, Errors, Duration. USE (for resources like CPU and disks): Utilization, Saturation, Errors.

Track latency as percentiles (p50, p95, p99), not averages.

Logging well

• Use structured logs (JSON) with consistent fields: timestamp, level, service, request ID or trace ID, user ID where allowed.
• Log events and decisions, not every line of code. Keep debug logs off in production by default.
• Never log secrets, passwords, tokens or full card numbers, and be careful with personal data.
• Ship logs to a central store with retention limits; logs get expensive fast.

Tracing

Each incoming request gets a trace ID, passed along in headers to every downstream call (the W3C traceparent header is the standard). Each service records spans: start time, duration, attributes and errors. OpenTelemetry is the vendor-neutral standard for producing metrics, logs and traces; all three major clouds accept it. At high traffic you sample traces, keeping all errors and slow requests and a fraction of normal ones.

Alerting and dashboards

• Alert on symptoms, not causes. Page someone when users are affected (SLO burn rate, error spike, latency over target), not when a single server's CPU is high.
• Every alert should be actionable, with a runbook link. Delete alerts nobody acts on.
• Dashboards: one overview per service with the golden signals, and deeper ones for debugging.

Using them together

An alert fires on a metric (error rate up). You open the dashboard to see when it started and which endpoint is affected. You pick a failing request's trace to find the slow or failing service, then read that service's logs for that trace ID to see the actual error.

Trade-offs

What you gain

• Faster detection and diagnosis, so shorter outages.
• Data to make decisions: capacity planning, SLOs, performance work.
• Traces make microservices understandable.

What it costs you

• Telemetry costs real money at scale, especially logs and high-cardinality metrics.
• Instrumentation takes effort and has to be kept up to date.
• Too many alerts and dashboards create noise and fatigue.

When not to use it

• Don't log everything “just in case”; sample and set retention.
• Don't add metric labels with unbounded values (like user ID) to metrics; put those in logs or traces.

In the cloud

Monitoring, logging and tracingEvery major cloud sells this as a managed service, so you configure it rather than build it.

AWS2

• Amazon CloudWatch (opens vendor docs)

  Metrics, logs, alarms and dashboards for AWS resources and your apps.

• AWS X-Ray (opens vendor docs)

  Distributed tracing. AWS now recommends OpenTelemetry instrumentation; the X-Ray SDKs are in maintenance mode.

Azure2

• Azure Monitor (opens vendor docs)

  Collects metrics and logs from Azure resources, with alerts and dashboards.

• Application Insights (opens vendor docs)

  Application performance monitoring and distributed tracing, built on OpenTelemetry.

Google Cloud3

• Cloud Monitoring (opens vendor docs)

  Metrics, uptime checks, dashboards and alerting.

• Cloud Logging (opens vendor docs)

  Stores, searches and analyzes logs.

• Cloud Trace (opens vendor docs)

  Distributed tracing for latency analysis.

Apps emit metrics, logs and traces (ideally with OpenTelemetry); dashboards and alerts sit on top.

AWSAzureGoogle Cloud

1. App with OpenTelemetry
2. CloudWatch metrics + logs
3. X-Ray traces
4. CloudWatch alarms

Interview questions

Try answering out loud first, then open the model answer.

What would you monitor for a web API, and what would you alert on?+

Model answer

Per endpoint: request rate, error rate (5xx and important 4xx), and latency at p50, p95 and p99; plus saturation signals like CPU, memory, connection-pool usage, queue depth and database load. Dependencies: their latency and errors. I'd define SLOs, for example 99.9% of requests succeed and 99% are under 300 ms, and alert on fast and slow error-budget burn rates. Resource alerts like disk almost full would go to a ticket queue, not a page.

Explain distributed tracing.+

Model answer

When a request enters the system it gets a trace ID. Each service creates spans for its work, recording start, duration, attributes and errors, and passes the trace context to downstream calls in headers. A tracing backend assembles the spans into a tree or timeline for the request, showing exactly which call took how long and where errors happened. It's essential in microservices, where one user action touches many services. Traces are usually sampled to control cost, keeping all errors and slow ones.

Latency is up but no alerts fired on any single service. How do you investigate?+

Model answer

Start from the user-facing metric and slice it by endpoint, region, customer and version to find what changed and when; correlate with deploys and config changes. Pull traces for slow requests and compare them with fast ones to see which span grew. Check shared resources like database load, connection pools, caches (hit rate dropped?), and noisy neighbors. Then add the missing SLI or alert so this is caught automatically next time.

Quiz

Pick an answer to see whether it’s right and why.

1. 01Users report that checkout is slow. Which tool shows which service in the chain is slow for a specific request?

   CPU metricsA distributed traceThe load balancer's access log
2. 02What makes a good alert?

   CPU over 80% on any serverUsers are being affected, for example the error rate is burning through the SLO's error budget too fastEvery warning in the logs
3. 03Why log in a structured format like JSON with a request ID?

   It's smallerSo logs can be searched and filtered by field, and all log lines for one request can be found togetherBecause text logs aren't allowed in the cloud
4. 04Which tells you fastest that something is wrong across all requests?

   MetricsA single log lineReading the code
5. 05Why sample traces instead of keeping every one?

   Traces can't legally be storedTracing every request costs a lot of storage and some overheadSampling makes traces more accurate
6. 06An alert fires on CPU above 80% several times a day, but users are fine. What's better?

   Raise the threshold to 99% CPUTurn off all alertsAlert on what users feel, like error rate or latency against the SLO

0 of 6 answered

Key terms

Observability

How well you can tell what's happening inside a system from its outputs: metrics, logs and traces.

Metrics

Numbers measured over time, like requests per second, error rate and latency.

Logs

Timestamped records of individual events, such as one failed request and why.

Distributed tracing

Following one request across every service it touches, with timing for each step.

SLO

Service level objective. The internal target, such as 99.9% of requests succeed each month.

Percentile (p50, p99)

The p99 latency is the time that 99% of requests beat. It shows how slow your slowest users' requests are.