Observability Engineering for Microservices: Prometheus 3, Grafana 13 & OpenTelemetry Tracing

Installing kube-prometheus-stack

Prometheus 3.0.0, released November 14, 2024, is the first major version in seven years. For Kubernetes deployments, kube-prometheus-stack (currently v86.2.2) is the de facto standard — a single Helm chart that installs Prometheus, Grafana, Alertmanager, kube-state-metrics, and node-exporter together as a cohesive, operator-managed unit.

```bash helm repo add prometheus-community https://prometheus-community.github.io/helm-charts helm repo update

helm install kps prometheus-community/kube-prometheus-stack \ --namespace monitoring --create-namespace \ --set prometheus.prometheusSpec.serviceDiscoveryRole=EndpointSlice \ --set prometheus.prometheusSpec.scrapeInterval=30s \ --set prometheus.prometheusSpec.evaluationInterval=30s ```

The serviceDiscoveryRole=EndpointSlice flag is critical. Without it, prometheus-operator defaults to the legacy endpoints role even on modern clusters — covered in the next section.

Kubernetes Service Discovery: endpointslice vs. endpoints

Prometheus discovers scrape targets through kubernetes_sd_configs. The role value controls which Kubernetes API it queries to enumerate targets.

The endpoints role uses the classic v1/Endpoints API — one object per Service, growing unbounded as pods scale. The endpointslice role uses discovery.k8s.io/v1 (GA since Kubernetes 1.21), which shards a Service's endpoints into slices of up to 100 each. Smaller watch events mean lower memory pressure in Prometheus and the API server as you scale past dozens of services.

prometheus-operator v0.76.0 (merged July 25, 2024) introduced the serviceDiscoveryRole field on the Prometheus CRD. When unset, it silently defaults to "Endpoints". For any cluster running Kubernetes 1.21 or later, set it explicitly to EndpointSlice. Prometheus 3.0 also removed support for the deprecated discovery.k8s.io/v1beta1 EndpointSlice API — clusters relying on the beta API will lose all scrape target discovery silently after upgrading to Prometheus 3.

Use the Service Discovery tab at http://localhost:9090/service-discovery to confirm that endpointslice discovery found the correct pod targets before they enter the active scrape pool.

node-exporter and kube-state-metrics

The stack ships two exporters that answer fundamentally different questions.

**node-exporter** is a DaemonSet — one pod per node. It reads from the host's /proc and /sys and exposes hardware and OS metrics with the node_ prefix: node_cpu_seconds_total, node_memory_MemAvailable_bytes, node_filesystem_avail_bytes. It has zero knowledge of Kubernetes objects. Query it to answer "Is this node's disk filling up?" or "Is a node's CPU saturated?"

**kube-state-metrics** (KSM) is a single Deployment. It watches the Kubernetes API server and synthesizes object-state metrics with the kube_ prefix: kube_pod_status_phase, kube_deployment_status_replicas_available. It answers "Are my pods running?" and "How many replicas are ready?" — but exposes nothing about actual CPU or memory consumption.

PromQL rate() and Counter Reset Handling

[rate(v[window])](https://prometheus.io/docs/prometheus/latest/querying/functions/) calculates the per-second average rate of increase for a counter over the specified time window. It handles counter resets — which happen on every pod restart — by detecting any decrease in value and treating it as a reset to zero, then computing pre-reset and post-restart trends independently before combining them into a single smooth average.

Consider an api-server pod restarting at 14:02 with http_requests_total dropping from 1,432,100 to 0. rate() detects the drop, computes separate pre- and post-reset segments, and returns a smooth per-second average for the full five-minute window. No spike, no NaN.

# CORRECT: rate first, then aggregate across instances
sum(rate(http_requests_total{job="api-server"}[5m])) by (instance)

Applying rate() after aggregation defeats reset detection — summing across pods first masks a single-pod reset inside the group total. Always call rate() before sum().

increase(metric[5m]) equals rate(metric[5m]) * 300. Both handle resets identically; the difference is units. Use rate() for alerting because its output is time-window agnostic. Use increase() for human-readable panels showing "requests in the last 5 minutes." For a 30-second scrape interval, set your minimum rate window to [2m] — four times the scrape interval — to guarantee at least two samples are always present in the window.

Recording Rules and promtool Validation

A recording rule pre-computes an expensive PromQL expression and writes the result as a new time series at each evaluation interval. Dashboards that would take four seconds to load eight separate sum(rate(...)) panels load in milliseconds after pre-computation.

groups:
  - name: api_request_rates
    interval: 60s
    rules:
      - record: job_path:http_requests_total:rate5m
        expr: |
          sum by (job, path) (
            rate(http_requests_total[5m])
          )

The naming convention `level:metric:operations` is a Prometheus best practice: job_path is the aggregation level, http_requests_total is the base metric, rate5m is the operation. The colon character is reserved exclusively for recording rule names — never use it in raw exporter metric names or application instrumentation.

Validate rules without starting Prometheus:

promtool check rules rules/request_rates.yaml
# Exit 0: success  |  Exit 1: syntax error  |  Exit 3: lint-fatal violation (--lint-fatal flag)

Recording rules use record:. Alerting rules use alert:, add a for: duration for the pending state, and support annotations:. promtool check rules validates both rule types with the same command; promtool test rules runs unit tests against expected output metric values.

Verifying Scrape Health in the Targets UI

After installing the stack, open http://localhost:9090/targets. Each row shows target state (UP/DOWN), last scrape time, scrape duration, and the error message when a target is DOWN. The `up` metric equals 1 on a successful scrape and 0 on failure, and is automatically populated for every scrape target.

Three PromQL queries to keep in your scrape-health runbook:

up == 0                                           # targets currently failing
scrape_duration_seconds > 10                      # approaching timeout (warn early)
scrape_samples_post_metric_relabeling > 5000      # cardinality spike detection

The cardinality query is particularly valuable at cluster onboarding: a newly deployed service that accidentally includes a user_id label on every metric will multiply TSDB size by an order of magnitude and first appear here, not in a billing alert.

Hands-on Exercise

Goal: Deploy kube-prometheus-stack on a local cluster and verify scrape health end-to-end.

1. Install into a kind or minikube cluster with serviceDiscoveryRole: EndpointSlice and scrapeInterval: 30s set in values.yaml.
2. Port-forward the Prometheus service: kubectl -n monitoring port-forward svc/kps-prometheus-prometheus 9090:9090.
3. Open http://localhost:9090/targets — every target should show State: UP.
4. Run up == 0 in the Expression Browser. Expect an empty result set.
5. Write a recording rule job:up:avg with expr: avg by (job) (up) in rules/health.yaml. Validate with promtool check rules rules/health.yaml and confirm exit code 0.
6. Apply the rule via a values.yaml additionalPrometheusRulesMap entry and reload with curl -X POST http://localhost:9090/-/reload. Query job:up:avg in the Graph view — it should return values near 1.0 for all jobs.

Success criteria: No up == 0 results; promtool exits 0; job:up:avg is queryable and returns expected values near 1.0.

02-promql-slos-and-alertmanager covers histogram_quantile() for SLO latency percentiles and multi-burn-rate alerting with Alertmanager.

Modeling Payment-Service Reliability with Latency Percentiles

A payment-path SLO covers three reliability dimensions: error rate (fraction of HTTP 5xx responses), latency percentiles (p99 < 500ms is the industry benchmark for checkout), and saturation (queue depth, thread-pool utilization). Together they answer: does the service keep its promises? Error rate tells you if requests succeed; latency tells you how well; saturation predicts when the next failure arrives.

p99 matters more than p95 on payment paths. A checkout that fails or times out means a lost transaction, not a slow page. Industry targets consistently land at p99 < 500ms for checkout services. When each percentile tick above that represents direct revenue exposure, you model at p99 or p99.9, not p95.

histogram_quantile() and the Case for Native Histograms

histogram_quantile(φ, b) estimates any quantile from a histogram. The call looks identical for classic and native histograms — the difference is in interpolation accuracy.

Classic histograms decompose into _sum, _count, and _bucket{le="..."} series with bucket boundaries fixed at instrumentation time. The function uses linear interpolation within whichever bucket contains the target quantile. If your true p95 is 220ms but your buckets are [100ms, 200ms, 300ms], the function returns 295ms — a 34% overestimate — because it treats the entire 200–300ms bucket as uniformly occupied. Histograms and summaries | Prometheus

Native histograms (stable since Prometheus 3.8.0) encode a single composite sample with dynamic exponential buckets. With NativeHistogramBucketFactor: 1.1, each bucket is roughly 4% wider than its predecessor. The same 220ms p95 value returns 228ms — a 4% error versus 34%. Native histograms also compress to roughly 8× smaller protobuf payloads when many buckets are active. Native Histograms | Prometheus

The accuracy gap matters on payment paths because a 100ms SLO margin is not a rounding error. Misidentifying a 220ms true latency as 295ms means your alert fires on histogram noise before the actual SLO boundary.

Multi-Window Multi-Burn-Rate SLO Alerting

A burn rate is how fast, relative to the SLO window, the service consumes its error budget. Burn rate 1 means the budget exhausts exactly by window end; burn rate 14.4 on a 30-day window exhausts it in roughly 50 hours. The Google SRE Workbook defines three alert tiers using this concept.

Single-threshold alerts fail in two opposite directions: a short window fires on transient spikes (false positives); a long window takes hours to reset after the incident resolves (alert fatigue). The multi-window multi-burn-rate (MWMBR) AND gate fixes both.

Tier 1 (page, 14.4× burn rate) requires both a 1-hour window AND a 5-minute window to simultaneously exceed the threshold. A 30-second traffic spike raises the 5m window but not the 1h window — no page fires. An actual outage raises both within minutes. After the fix, the 5m window drains in under five minutes, cutting reset time from hours to minutes.

- alert: CheckoutHighErrorBurnRate
  expr: |
    (sum(rate(checkout_requests_total{status=~"5.."}[1h]))
     / sum(rate(checkout_requests_total[1h])) > 0.01440)
    and
    (sum(rate(checkout_requests_total{status=~"5.."}[5m]))
     / sum(rate(checkout_requests_total[5m])) > 0.01440)
  for: 2m
  labels:
    severity: page
    team: payments
  annotations:
    summary: "Checkout error budget burning at >14.4× — both 1h and 5m windows exceeded"

Tier 2 uses a 6× burn rate over 6h + 30m windows (fires after ~5% of monthly budget consumed). Tier 3 uses 1× burn rate over 3d + 6h windows for slow-burn detection. All three tiers together provide complete coverage: acute outages hit Tier 1 within ~5 minutes; slow degradations accumulate into Tier 3 over hours.

The latency SLO alert follows the same AND-gate pattern, replacing the error ratio with a histogram_quantile() expression evaluated over two windows.

Alertmanager Routing Tree and Inhibition Rules

The Alertmanager routing tree routes firing alerts to receivers (PagerDuty, Slack, email) based on label matchers. The root route matches all alerts; child routes refine by label. Three timing parameters govern notification behavior with distinct semantics:

• `group_wait` (default 30s): hold time before sending the first notification for a new alert group, allowing co-firing alerts to arrive and be batched
• `group_interval` (default 5m): after the initial notification, how long before re-checking whether new alerts have joined the group
• `repeat_interval` (default 4h): how long before re-sending for an unchanged active group — the nag timer

Inhibition rules suppress target alerts while a source alert is active. During a full checkout outage, you want the page to fire cleanly — not alongside a flood of redundant warning-level alerts for the same service:

inhibit_rules:
  - source_matchers: [severity=page, team=payments]
    target_matchers: [severity=warning, team=payments]
    equal: ['alertname']

The equal field is critical: without it, any page alert in any cluster silences all warning alerts globally. For multi-cluster deployments, always scope with equal: ['cluster', 'namespace'].

PrometheusRule CRD and promtool Unit Tests

In Kubernetes-native setups, alerting rules live in PrometheusRule CRDs (kind: PrometheusRule, apiVersion: monitoring.coreos.com/v1). The prometheus-operator watches these resources and injects matching rules into any Prometheus instance whose spec.ruleSelector labels match the CRD's labels. A missing label — e.g., omitting release: prometheus-stack — causes the rule to be silently ignored: no error, no alert. Always confirm the resource loaded with kubectl get prometheusrule -n payments.

Before applying, validate with promtool:

promtool check rules checkout_slo_rules.yaml          # syntax only
promtool test rules tests/checkout_slo_test.yaml      # full unit test

Unit test files supply input_series (mock counter increments), evaluation_interval, and alert_rule_test blocks asserting which alerts should fire at each eval_time. A test simulating a 1-minute error spike should assert exp_alerts: [] for the Tier 1 alert — the 1h long window dilutes the single-minute spike below the 14.4× threshold.

Diagnosing Alerts with the Alertmanager API

When a page doesn't arrive or routes to the wrong receiver, the Alertmanager v2 API is the first diagnostic stop:

```bash # Show all currently active alerts curl http://alertmanager:9093/api/v2/alerts?active=true | jq .

amtool config routes test prints the matched receiver without sending a real notification — run this in CI and before on-call handoffs. Alertmanager 0.33.0 also accepts receiver_matchers filters on /api/v2/alerts for targeted diagnostics by receiver name.

Hands-On Exercise

Deploy a complete payment-path SLO alert stack against a local checkout service:

1. Instrument a checkout service with NativeHistogramBucketFactor: 1.1 (Go SDK) and confirm histogram_quantile(0.99, ...) returns a value against live traffic.
2. Write checkout_slo_rules.yaml with the Tier 1 (14.4×, 1h+5m) and Tier 2 (6×, 6h+30m) alert expressions from this chapter.
3. Run promtool check rules checkout_slo_rules.yaml — fix any reported syntax errors.
4. Write a unit test that (a) simulates a 2% error rate for 60 minutes and asserts Tier 1 fires at eval_time 62m, and (b) simulates a 1-minute spike and asserts Tier 1 does not fire. Run promtool test rules — both test cases must pass.
5. Apply a PrometheusRule CRD with release: prometheus-stack label; confirm with kubectl get prometheusrule -n payments.
6. Add an inhibition rule that suppresses warning alerts while a page fires for the same alertname. Run amtool config routes test for a payment page alert and confirm the receiver is payments-pagerduty.

Success criteria: promtool test rules exits 0 with "SUCCESS: N tests passed"; amtool config routes test prints payments-pagerduty as the resolved receiver.

Next: 03-grafana-13-dashboards-and-git-sync builds on these Alertmanager rules by adding Grafana-managed alert rules, contact points, and Git Sync for dashboard versioning.

Provisioning Prometheus as a Data Source via YAML

Grafana's provisioning system reads YAML files at startup and on every reload, making data source configuration repeatable and auditable without a single UI click. Drop a datasources.yaml file into provisioning/datasources/ and Grafana wires the connection — the source of truth stays in your Git repository, not Grafana's database.

A production-ready Prometheus configuration:

# provisioning/datasources/prometheus.yaml
apiVersion: 1
prune: true
datasources:
  - name: Prometheus
    type: prometheus
    uid: prometheus-main
    url: http://prometheus:9090
    isDefault: true
    editable: false
    jsonData:
      scrapeInterval: "15s"
      httpMethod: "POST"

Two fields are load-bearing. uid: prometheus-main creates a stable identifier that every dashboard panel references in its datasource.uid field. Omit it and Grafana auto-generates a UID from the name; rename the data source and every dependent panel breaks silently. editable: false greys out the settings in the UI so no one can drift the connection away from the declared configuration. The top-level prune: true makes deletions declarative: remove the stanza from YAML and Grafana deletes the data source on the next reload, no manual UI cleanup required.

Building the Dashboard JSON Model

Every Grafana dashboard is a JSON document. The root object carries metadata (title, uid, tags, refresh), a templating block for variables, and a panels array. Each panel has a type, a gridPos object positioning it on the 24-column grid, and type-specific options and fieldConfig blocks.

Choose the panel type based on the shape of data you need to render:

The fieldConfig.defaults.unit and thresholds fields apply uniform formatting across all series. A Gauge panel colors a payment success rate red below 99% with no conditional logic — the threshold steps carry it.

Template Variables for Dynamic Dashboards

Template variables turn a static dashboard into a multi-service explorer. A query variable executes PromQL against your data source to populate its dropdown:

label_values(http_requests_total, service)

Set refresh to "On dashboard load" so new services appear automatically as your workload grows. A custom variable uses a static comma-separated list — exactly right for an environment selector (prod,staging,dev) whose options do not change with your workload.

Reference variables in panel queries as $service or ${service}. Chain them when one variable constrains another: a $service query whose PromQL reads label_values(up{env="$env"}, service) shows only services active in the chosen environment, eliminating phantom entries from decommissioned workloads.

Template variables and repeated panels solve different problems. A template variable with multi-select parameterizes all panels simultaneously — right when a viewer wants to focus on one service at a time. Panel repetition auto-generates N copies of a panel, one per variable value, all visible at once — right when the goal is side-by-side comparison across all services. Mixing both on the same dashboard usually signals it needs to be split by purpose.

Git Sync: Dashboards as Code

Grafana 13.0 (April 21, 2026) promoted Git Sync to General Availability across Cloud, Enterprise, and OSS editions. Git Sync connects Grafana to a GitHub, GitLab, or Bitbucket repository and keeps dashboards synchronized bidirectionally: UI edits commit back to Git; Git merges propagate back to Grafana.

The PR workflow turns every change into a reviewable event: an engineer edits a panel and clicks Save; Grafana commits the JSON to a feature branch and opens a PR automatically; a reviewer merges it; Git Sync detects the merge within 5 seconds via webhook (or 30 seconds via polling) and updates the live dashboard.

The structural risk is merge conflicts. A dashboard has no sub-file granularity — all panels, variables, and layout live in one JSON object, so any two concurrent edits touch the same file. The PR workflow mitigates this by serializing changes through review, but the root cause remains: one editor per dashboard at a time. Per Observability as Code with Grafana Git Sync, start with a small subset; performance degrades above 200 dashboards.

Grafana-Managed Alert Rules and the Alert State Machine

Grafana's unified alerting engine offers two architectures. Data-source-managed rules are stored and evaluated by the data source's ruler component — available only for Mimir and Loki — and are the right choice when horizontal evaluation scalability matters more than multi-condition logic. Grafana-managed rules are stored in Grafana's own database, evaluated by the Grafana server, and support any connected data source plus multi-query expressions using Reduce, Resample, and Math operators.

Build a multi-condition rule that fires if P99 latency exceeds 500 ms or error rate exceeds 1%:

• Query A: histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket{service="payments"}[5m])) by (le))
• Query B: sum(rate(http_requests_total{status=~"5..",service="payments"}[5m])) / sum(rate(http_requests_total{service="payments"}[5m]))
• Math expression C: $A > 0.5 || $B > 0.01

Assign the rule to an evaluation group and set the Pending period to 2–5 minutes for any signal that can spike transiently. The Pending period is a hysteresis buffer: the condition must breach continuously for the full duration before the alert transitions to Firing. Setting it to zero fires on the first evaluation breach — right only for signals where every false positive is acceptable. The Grafana Labs 4th Annual Observability Survey found 30% of SREs cite alert fatigue as their biggest obstacle; a nonzero pending period is the cheapest mitigation.

Attach a contact point to route notifications when the alert fires. Contact points bundle one or more integrations — Grafana ships over 25 types including Slack, PagerDuty, Microsoft Teams, email, and webhook — into a single named destination. A single contact point can aggregate several integrations, so a critical alert simultaneously pages on-call via PagerDuty and posts a Slack summary in one delivery step.

Hands-On Exercise

Goal: Provision a Prometheus data source, build a two-panel dashboard with template variables, connect Git Sync, and observe a multi-condition alert cycle through the full state machine.

Steps:

1. Create provisioning/datasources/prometheus.yaml with the YAML from this chapter. Restart Grafana and verify the data source appears under Configuration → Data sources with a "Managed" badge and the URL field greyed out.

1. Open Dashboards → New Dashboard → JSON model, paste the dashboard skeleton, and confirm the $service and $env dropdowns populate.

1. Edit the P99 panel title to P99 Latency — $service ($env) and click Save. Verify a PR appears in your linked repository within 30 seconds. Merge it and confirm the title updates live in Grafana.

1. Navigate to Alerting → Alert rules → New alert rule. Create a Grafana-managed rule with Query A (P99), Query B (error rate), and Math expression C as above. Set the pending period to 2m. Attach a contact point wired to a test Slack webhook. Temporarily lower the threshold to $A > 0 to force a breach. Watch Normal → Pending → Firing in the state view and confirm a Slack notification arrives.

Success criteria: - Data source shows "Managed" badge; connection test passes. - Git PR was opened on save; dashboard reflected the merged title change within 30 seconds. - Alert transitioned through Pending and reached Firing within 2 minutes; Slack notification received.

Next, we instrument the services whose metrics you just charted and route their distributed traces into a backend: 04-opentelemetry-sdk-and-jaeger-v2.

Auto-Instrumenting Your Services Without Code Changes

The fastest path to traces is zero-code instrumentation: the OTel SDK intercepts your framework's HTTP, database, and queue calls at runtime without any changes to application source files.

For Node.js, install two packages and set three environment variables:

```bash npm install --save @opentelemetry/api @opentelemetry/auto-instrumentations-node

OTEL_SERVICE_NAME=payment-service \ OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://localhost:4317 \ OTEL_TRACES_EXPORTER=otlp \ node --require @opentelemetry/auto-instrumentations-node/register app.js ```

For Python, opentelemetry-bootstrap scans your installed packages, installs matching instrumentation libraries, and the opentelemetry-instrument wrapper monkey-patches them before your first import — no code changes required:

```bash pip install opentelemetry-distro opentelemetry-exporter-otlp opentelemetry-bootstrap -a install

OTEL_SERVICE_NAME=inventory-service \ OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://localhost:4317 \ opentelemetry-instrument python app.py ```

OTEL_SERVICE_NAME sets the service.name resource attribute on every span. Without it, Jaeger labels all traces unknown_service and your RED dashboards collapse every service into one unfiltered noise bucket. This is the single most important variable to get right before anything else. See the Node.js and Python zero-code guides for the full list of supported frameworks.

The OTel Collector Pipeline: Receive, Sample, Export

The OTel Collector sits between your SDKs and Jaeger, running a three-stage pipeline: receivers → processors → exporters. The production configuration for a traces pipeline is:

```yaml # otel-collector-config.yaml receivers: otlp: protocols: grpc: endpoint: 0.0.0.0:4317 http: endpoint: 0.0.0.0:4318

processors: memory_limiter: check_interval: 1s limit_mib: 1000 spike_limit_mib: 200 tail_sampling: decision_wait: 30s num_traces: 50000 policies: - name: capture-errors type: status_code status_code: status_codes: [ERROR] - name: capture-slow-traces type: latency latency: threshold_ms: 500 batch: send_batch_size: 8192 timeout: 200ms

exporters: debug: verbosity: detailed # remove in production otlp/jaeger: endpoint: jaeger:4317 tls: insecure: true

service: pipelines: traces: receivers: [otlp] processors: [memory_limiter, tail_sampling, batch] exporters: [debug, otlp/jaeger] ```

The otlp/jaeger exporter name is intentional — the legacy jaeger exporter was removed from the OTel Collector at v0.85.0. Teams porting v1 playbooks who still write exporters: jaeger: will see unknown exporter type: jaeger at startup. Replace it with an otlp/jaeger block pointing at Jaeger's port 4317.

For the SDK-to-Collector transport, prefer OTLP/gRPC (port 4317) when both are in the same cluster: persistent multiplexed HTTP/2 connections reduce per-batch latency compared to OTLP/HTTP. Use OTLP/HTTP (port 4318) when crossing a firewall that blocks non-standard ports, or when you need JSON-encoded spans for debugging with standard HTTP tools.

The debug exporter (named logging before Collector v0.86.0) prints full span payloads to stdout. Use it to confirm the Collector is receiving spans before Jaeger is even running, then remove it in production.

Tail-Based Sampling: Guaranteeing Error and Slow-Trace Capture

Head-based sampling makes its decision at trace initiation, before a single span is emitted. At 5% head sampling, 95% of your traces are gone before you know which ones contained errors or breached your latency SLO. For a fintech payment platform with a 0.3% error rate and a compliance requirement to capture every failure, head sampling at any rate below 100% is the wrong tool.

Tail-based sampling defers the decision to the OTel Collector, which buffers incoming spans for the decision_wait window (default 30 seconds) and evaluates policies against the complete trace. The two canonical SRE policies are:

• `status_code: [ERROR]` — retains any trace where at least one span carries ERROR status, guaranteeing 100% error capture.
• `latency: threshold_ms: 500` — retains any trace whose end-to-end duration exceeds 500 ms, capturing every p99 SLO breach with early-onset headroom.

The memory cost is real. According to a practitioner sizing analysis, 1,000 traces/second with a 15-second decision_wait and 10 spans per trace at 1 KB each requires roughly 150 MB of span buffer and at least 500 MB total — with 1 GB recommended to include overhead and safety margin. At 5,000 traces/second with the same wait, the buffer alone hits 2.4 GB. At that scale, adding a 5% probabilistic head-sampling pre-filter at the SDK layer reduces what reaches the tail sampler without sacrificing error coverage.

Deploying Jaeger v2 and Its OTLP-Native Architecture

Jaeger v2 was released by the CNCF on November 12, 2024. The most visible change: four separate binaries (agent, collector, ingester, query) collapsed into one — jaegertracing/jaeger:2.x — configured entirely via a YAML file.

Jaeger v2 is built on the OTel Collector framework, so it uses the same receiver/processor/exporter pipeline model as the standalone Collector. It accepts OTLP natively on port 4317 (gRPC) and 4318 (HTTP) with no translation layer between the wire format and internal storage. The jaeger-agent DaemonSet that teams ran as a sidecar in v1 is eliminated; the OTel Collector (or a direct SDK OTLP export) replaces its role entirely.

services:
  jaeger:
    image: jaegertracing/jaeger:2.19.0
    command: ["--config", "/jaeger/config.yaml"]
    volumes:
      - ./jaeger-config.yaml:/jaeger/config.yaml
    ports:
      - "4317:4317"   # OTLP gRPC
      - "4318:4318"   # OTLP HTTP
      - "16686:16686" # Jaeger UI
      - "8888:8888"   # Prometheus-compatible metrics

A critical migration trap: Jaeger v2 does not accept v1 environment variables like COLLECTOR_OTLP_ENABLED or SPAN_STORAGE_TYPE. Teams porting Kubernetes manifests that set these env vars will get silent defaults or startup errors. The Jaeger v2 Deployment docs document the YAML equivalents for each former env var.

Connecting these traces to your Prometheus dashboards via Grafana Explore and exemplar click-through is covered in 05-correlating-metrics-traces-and-alerts.

W3C traceparent and Cross-Service Context Propagation

Every OTel SDK injects and extracts the traceparent HTTP header on outbound and inbound requests. The W3C Trace Context specification defines four fields in that header:

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
             ^^                                                        version (always 00)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                      trace-id (32 hex = 16 bytes)
                                                ^^^^^^^^^^^^^^^^      parent-id (16 hex = 8 bytes)
                                                                 ^^   trace-flags (01 = sampled)

When Service A calls Service B, Service A's SDK injects this header into the outbound HTTP request. Service B's SDK extracts the trace-id and parent-id, creates a child span under the same trace-id, and sets its parent span ID to Service A's span ID. Jaeger then renders both spans in one unified trace tree, letting you trace the full call chain across service boundaries in a single view.

Diagnosing Context Propagation Failures

The symptom of a broken propagation chain is unmistakable: instead of one trace tree spanning multiple services, Jaeger shows disconnected root spans — Service B appears to have started its own independent trace rather than a child of Service A.

The most common production cause is a proxy or API gateway stripping traceparent before it reaches the downstream service. Nginx, Envoy, and some managed API gateways drop headers they do not recognize by default. Fix: explicitly add traceparent and tracestate to your proxy's header pass-through allowlist. To verify, enable the OTel Collector debug exporter on the receiving service and confirm that incoming spans carry a trace_id matching the upstream service — if the trace_id is fresh on every request, the header is being stripped upstream.

Hands-On Exercise: End-to-End Trace Pipeline

Goal: Emit a trace from a Node.js service and verify it appears in Jaeger v2 with the correct service name.

1. Start Jaeger v2 in Docker: docker run -p 4317:4317 -p 16686:16686 jaegertracing/jaeger:2.19.0
2. Start the OTel Collector with the full config above, setting endpoint: localhost:4317 in the otlp/jaeger exporter block.
3. Launch your Node.js app with OTEL_SERVICE_NAME=payment-service and the --require @opentelemetry/auto-instrumentations-node/register flag.
4. Make one HTTP request to the app to trigger a trace.
5. Open http://localhost:16686, select payment-service from the Service dropdown, and click Find Traces.

Success criteria: - At least one trace appears under payment-service in the Jaeger UI — not unknown_service. - The trace shows at least one span with an HTTP method and status code. - The OTel Collector stdout (from the debug exporter) shows the span payload before the trace appears in Jaeger, confirming the pipeline is live end-to-end.

Next chapter: 05-correlating-metrics-traces-and-alerts connects these Jaeger traces to your Prometheus RED metrics via exemplars and Grafana Explore, completing the three-pillar observability loop.

What Are Exemplars and Why They Bridge the Gap

Metrics tell you something is wrong. Traces tell you why. The missing link is the exemplar — a trace_id embedded directly inside a Counter or Histogram bucket sample. When your p99 latency time series spikes at 14:12, the exemplar stored alongside that data point carries the exact trace ID of a representative slow request. One click in Grafana Explore jumps from the spike to the full distributed trace without any manual log-grepping or time-range searching in Jaeger.

The OpenMetrics 1.0 Specification defines an exemplar as a label set (at minimum trace_id) plus a numeric value, appended inline to Counter and Histogram bucket samples. Gauges and summaries cannot carry exemplars. The spec enforces a hard 128-character limit on the combined label set — a 32-hex trace ID plus a 16-hex span ID plus both label names sums to roughly 58 characters, comfortably within the limit.

Enabling Exemplar Scraping in Prometheus

Prometheus ships with exemplar storage disabled. Enable it with the --enable-feature=exemplar-storage startup flag, then configure the circular buffer capacity in prometheus.yml:

```yaml storage: exemplars: max_exemplars: 100000 # ~10 MB at 100 bytes/exemplar

scrape_configs: - job_name: "payment-svc" scrape_protocols: - OpenMetricsText1.0.0 - OpenMetricsText0.0.1 - PrometheusText1.0.0 static_configs: - targets: ["payment-svc:8080"] ```

The scrape_protocols list is not optional detail. Without it, Prometheus negotiates the legacy text format and silently discards every exemplar at scrape time — no error, no warning, just missing data when you need it most. Verify exemplars are actually arriving before an incident forces you to find out they weren't:

curl -g 'http://localhost:9090/api/v1/query_exemplars?query=http_request_duration_seconds_bucket&start=<start>&end=<end>'

An empty data array means the wrong content-type was negotiated. Check the raw Content-Type response header on the service's /metrics endpoint.

Connecting Grafana to Jaeger

With Jaeger v2 already deployed (chapter 4), add the Jaeger data source via a provisioning YAML so the configuration is version-controlled and idempotent:

# grafana/provisioning/datasources/jaeger.yaml
apiVersion: 1
datasources:
  - name: Jaeger
    type: jaeger
    url: http://jaeger-query:16686
    access: proxy
    jsonData:
      tracesToMetrics:
        datasourceUid: prometheus-uid    # must match your Prometheus DS uid
        spanStartTimeShift: "-1m"
        spanEndTimeShift: "1m"
        tags:
          - key: "service.name"
            value: "service"
        queries:
          - name: "Request rate"
            query: "rate(http_requests_total{service=\"$__tags.service\"}[5m])"

The tracesToMetrics block enables bidirectional navigation: from a Jaeger span back to the Prometheus panel for the same service. The inverse direction — metric to trace — requires the Prometheus data source to also reference the Jaeger UID in an Exemplar section. Missing either linkage silently breaks one direction of the workflow.

Incident Triage: From Alert to Root Cause in Minutes

The payment degradation scenario ties all three pillars together. At 14:12 IST, payment-svc v2.3.1 deployed with a missing index on account_id. The p99 latency climbed from 145 ms to 4,200 ms within minutes. PaymentSvcFastBurn fired when both the 1-hour and 5-minute burn-rate windows simultaneously exceeded 14.4×.

The on-call SRE opened Grafana Explore and queried histogram_quantile(0.99, rate(http_request_duration_seconds_bucket{job="payment-svc"}[5m])). An exemplar ◆ star was overlaid on the p99 spike at 14:12. Hovering revealed trace_id="a3f8b1c92d45e670f9812345abcd6789". One click opened the full 6-span Jaeger trace in a split pane: the db-query child span — payment-svc → postgres-primary — accounted for 3,847 ms of the 4,187 ms root span. The db.statement attribute exposed an unindexed full-table scan on account_id.

From alert fire to root cause: 4 minutes — versus the 15–30 minutes a manual Jaeger search typically requires.

Multi-Burn-Rate Alert with Runbook Annotation

The PaymentSvcFastBurn rule applies the Google SRE Workbook multi-burn-rate methodology — built on the ch2 alerting foundation — to this live incident:

- alert: PaymentSvcFastBurn
  expr: |
    (
      job:payment_errors_per_request:ratio_rate1h > (14.4 * 0.001)
    and
      job:payment_errors_per_request:ratio_rate5m > (14.4 * 0.001)
    )
  labels:
    severity: page
  annotations:
    summary: "Payment SVC fast error budget burn (14.4x over 1h)"
    runbook_url: "https://runbooks.internal/payment-svc/high-error-rate"

The and between long and short windows is load-bearing logic. A single-window rule at 14.4× would page on any 1-minute transient spike that has already resolved. Requiring both windows simultaneously confirms the elevated burn rate is still ongoing at evaluation time, not historical. The runbook_url annotation is surfaced by Alertmanager in the notification payload — the on-call engineer can open the playbook before reaching the Grafana dashboard.

The three tiers serve distinct failure modes: 14.4× / 1h catches complete outages; 6× / 6h catches partial degradations; 1× / 3d tickets low-level error creep that would drain the budget invisibly across deployment cycles.

Tail-Sampling Threshold Tradeoff

Chapter 4 established a 500 ms threshold_ms for the tail-sampling latency policy. A seemingly attractive optimization is to raise it to 1,000 ms to reduce trace storage costs. Here is the analysis.

With payment-svc generating ~8,000 spans per minute (p50=120 ms, p90=380 ms), a 500 ms threshold retains roughly 12% of traces — about 960 per minute. Raising to 1,000 ms would reduce retained volume by an estimated 35%, saving roughly 1.8 GB per month.

The tradeoff is coverage loss in the 500–999 ms zone. During the v2.3.1 incident, the degradation ramp-up from 14:12 to 14:15 produced traces in the 600–900 ms range before the 4,200 ms peak appeared. At a 1,000 ms threshold, those early-onset traces would have been dropped — and with them, the exemplars that would have guided the investigation.

Decision record (retain 500 ms): The 35% storage saving (~1.8 GB/month) does not justify losing the 500–999 ms evidence window, which is precisely where gradual degradations appear before an SLO alert fires. Document this decision inline in the tail_sampling config as a comment referencing the incident date and this analysis.

Three-Pillar Observability Synthesis

Metrics, traces, and alerts each solve part of the incident workflow: metrics give continuously evaluated aggregate signals for SLO evaluation, traces give surgical root-cause evidence, and alerts make the system proactive. Exemplars bridge all three — a trace_id embedded in a Counter or Histogram sample links the metric spike to the exact causative request, converting a reactive page into a directed investigation.

Hands-On Exercise

Objective: Reproduce the full exemplar click-through workflow against a local payment-svc simulation and produce a written tail-sampling decision record.

Steps:

1. Start Prometheus with --enable-feature=exemplar-storage and the scrape_protocols block targeting a local service exposing OpenMetrics exemplars on /metrics.
2. Verify exemplar ingestion: run the query_exemplars curl command and confirm data is non-empty with trace_id labels present.
3. Configure the Jaeger data source in Grafana using the provisioning YAML above, with tracesToMetrics pointing to your Prometheus data source UID.
4. In Grafana Explore, query histogram_quantile(0.99, rate(http_request_duration_seconds_bucket{job="payment-svc"}[5m])) during a simulated latency spike. Confirm a ◆ marker appears on the graph.
5. Click the ◆ exemplar and verify the "Query with Jaeger" button appears and opens the trace in a split pane.
6. Write a ≤200-word decision record justifying whether to set threshold_ms to 500 ms or 1,000 ms for your service's observed latency distribution. Address both the storage saving and the coverage tradeoff explicitly.

Success criteria: The exemplar ◆ is visible in Explore, clicking it opens a Jaeger trace in split-pane view, and your decision record names a specific latency zone that would be lost at the higher threshold.