Two Samplers, Opposite Bets

Two on-call stories, from two different teams.

The platform engineer’s tracing bill is enormous, so someone turned sampling down to 10%. Then a real incident hits — and the one trace that would have explained it was thrown away. The alternative, keeping everything, is how finance ended up in the reliability review.

The ML engineer’s LLM feature is flaky in a way that breaks evaluation. The same prompt returns different answers, a regression shows up, and nobody can say whether the model changed or the config did — because no one logged the temperature.

These look like unrelated problems solved by unrelated tools. They’re the same decision: given a probability distribution over outcomes, which ones do you keep or emit, and how do you deliberately bias that choice? And they’re governed by the same quantity.

The theory

Start with one definition. The self-information, or surprise, of an outcome x with probability p(x) is

I(x) = −log₂ p(x)

Rare outcomes carry more information; certain ones carry none. The average surprise of a whole distribution is its Shannon entropy:

H(p) = E[I(x)] = −Σ p(x) · log₂ p(x)

Everything below is a decision denominated in that currency.

Tail sampling (observability). Traces are not equally informative. A fast 200 OK is high-probability, low-surprise, and almost perfectly redundant with the thousands like it. An error, or a p99.9 latency outlier, is low-probability and high-surprise. The information carried by a trace is −log p. So if you must cut volume, the move that preserves the most information per byte is to drop the low-surprise mode and keep the high-surprise tail. That’s importance sampling: sample the rare-but-important at a higher rate than the common-but-redundant.

Temperature and top-k (AI). A language model produces a next-token distribution by softmax over logits zᵢ, with a temperature T:

pᵢ = exp(zᵢ / T) / Σⱼ exp(zⱼ / T)

Temperature is an entropy knob. As T → 0, mass collapses onto the top token and H → 0 (deterministic, greedy). As T → ∞, the distribution flattens toward uniform and H → log₂ n (maximum). Top-k and nucleus (top-p) sampling do the complementary thing: they truncate the low-probability, high-surprise tail before sampling.

So both systems are making a decision on the same axis — how much surprise (−log p) sits in what they keep or emit. The twist is the direction:

Telemetry :  MAXIMIZE retained surprise per byte   → keep the tail
Generation:  CONTROL emitted surprise (often ↓)     → suppress / tune the tail

Same currency, opposite targets — and for symmetric reasons. In telemetry, surprise is the expensive thing to lose: the rare trace is the whole reason you’re paying for traces. In generation, surprise is often the expensive thing to have: when a path needs to be reliable, a fat output tail is a liability. (Flip the sign for creative work, where you raise T to buy diversity.)

The application

Observability side. This is exactly what an OpenTelemetry tail_sampling processor encodes — policies that keep the informative tail and baseline-sample the redundant mode:

policies:
  - keep-errors:  status_code = ERROR   → 100%
  - keep-slow:    latency > 1s          → 100%
  - baseline:     probabilistic         →  10%

The reframe that changes how you tune it: you are not “keeping 10% of traces.” You are keeping ~all of the information at ~10% of the bytes — provided your policy’s notion of “the tail” actually matches where the surprise is.

AI side. The same idea lives in three request parameters — and in the OTEL GenAI semantic conventions that already name them:

gen_ai.request.temperature = 0.2
gen_ai.request.top_k        = 40
gen_ai.request.top_p        = 0.95

These set the entropy regime of every generation. If you don’t record them alongside the output, your latency and quality metrics are uninterpretable — T = 0.2 and T = 1.2 on the same endpoint are different systems, and comparing them is comparing two distributions you failed to label.

What to instrument, and what to do

• Observability: set tail-sampling policy by information value, not a flat percentage. Keep errors and slow traces at ~100%; baseline-sample the rest. Track information retained, not just percent kept. And watch the real failure mode: you cannot sample toward a kind of surprise you haven’t defined — a new class of high-surprise event (a novel error signature, a new slow dependency) is invisible until a policy names it.
• AI: log temperature, top_k, and top_p as first-class span attributes on every generation. Match the entropy regime to the path — low T for tool-calling and structured output, higher only where diversity is the goal — and instrument the cost of raising it: a higher temperature means a fatter output tail, which is directly harder to evaluate and monitor.
• Both: name, out loud, which direction you’re optimizing on the −log p axis. Most teams do it implicitly, and then are surprised by either the bill or the nondeterminism.

The two problems that opened this piece are the same problem, read in a mirror. One system keeps surprise because losing it is expensive; the other suppresses surprise because having it is expensive.

Same currency — −log p — opposite bets. Decide which one you’re making before your sampler decides for you.

If you want this mapped to your own stack — tracing pipeline or AI integration — get in touch.