Distributed Trace Hunt

How Distributed Trace Hunt Works

In distributed systems, partial traces are common. This recipe builds a repeatable

trace-driven debugging path: define the journey → validate propagation → close gaps.

Triggers

• Traces contain only some services (missing spans)
• You can't follow a single request end-to-end
• Production incidents require correlating logs/metrics/traces quickly

Steps

1. Choose one representative request journey and record its expected service chain.
2. Verify propagation headers and context injection/extraction per hop.
3. Check sampling policy and exporter/backpressure settings.
4. Add a "trace assertion" in a staging smoke test: request should produce spans in all hops.
5. Create a playbook for "missing spans" triage: sampling vs instrumentation vs async boundaries.

Expected Outcome

• You can trace a request through the intended service chain reliably.
• Incidents become faster to debug and less dependent on tribal knowledge.

Example Inputs

• "Our traces show the API gateway and service A, but nothing after."
• "Async job continuation loses the trace context."
• "We need to correlate frontend RUM with backend traces."

Tips

• Treat broken propagation as a correctness bug, not a logging preference.