Search traces and telemetry

Four questions, four tools. Why did this rank here? — for one search, per document: SearchTrace below, and Explain in Ranking. What did this one query cost? — the same search, counted rather than explained: SearchCosts. How long, how many candidates, which stage is slow? — across all searches in production: RetrievalTelemetry. Does the whole fleet agree, or is one retriever insisting? — for a fused page: RetrievalAgreementAnalyzer. Plus a score-to-confidence mapping for gating a result set you are not sure about.

All of it is opt-in and dependency-free. A null telemetry, or a search without a trace, allocates and records nothing.

Ranking trace (`SearchTrace`)

Explain breaks down one scorer’s arithmetic. SearchTrace goes further: it records every stage a document passed through, so a final rank can be walked back stage by stage — which engine scored it, what each merger contributed, what the reranker changed, what the boost did, and which route ran.

using LexiSharp.Core;

var trace = new SearchTrace();
var hits = engine.Search("refresh token", new SearchOptions(Limit: 5, Trace: trace));

foreach (var step in trace.Steps)
    Console.WriteLine($"{step.Stage,-6} {step.DocumentId,-8} {step.Before,8:0.000} -> {step.After,8:0.000}  {step.Detail}");

// route   d1              0.800 ->            dense (fallback: below threshold or no opinion)
// score   d1      4.819997 ->      4.819997  BM25
// merge   d1      0.016393 ->      0.016393  lexical=4.819997 dense=0.812003
// rerank  d1      4.819997 ->      0.912000
// boost   d1      0.912000 ->      1.824000  x2 +0

SearchOptions.Trace defaults to null, and a null trace is inert: no engine records, allocates or formats anything. Pass one and each stage appends to it as it runs, so a decorated pipeline accumulates its whole chain.

The merge’s own arithmetic

A weighted merge turns two raw scores into one with Σ wᵢ·sᵢ/divisorᵢ, and the per-document rows above carry only sᵢ. So a merger that can explain itself — WeightedScoreResultMerger does — also records one row per search, with an empty document id:

// merge              NaN -> NaN  weights lexical=1 dense=1 | max lexical=6.2 dense=0.812 | pools lexical=50 dense=50

The row comes before the documents it explains, is labelled by the same source names the per-document detail uses (sorted, so two runs produce the same string), and turns a contribution into a checkable number: 1·4.82/6.2 + 1·0.81/0.812 reproduces the merged score rather than inviting you to trust it. A rank merger does not implement IMergeExplainer and records no such row — it reads order, never multiplies a score, and reporting weights it never applied would be a more confident lie than an absent row.

Measured with HybridTraceBenchmarks, two engines over 2 000 documents: the untraced path allocates exactly the same bytes with and without this row (60.1 KB per search), because nothing below the trace is null guard executes; the traced path allocates 1.4 KB more (71.0 → 72.4 KB), one step and one string per search against the ten already written per document.

Cost — what is measured and what is not. Measured with BenchmarkDotNet’s MemoryDiagnoser and stable across repeated runs: a saturated trace allocates 0 additional bytes per search (1.34 KB either way), and a trace created per search costs ~0.47 KB (the object plus its backing array).

A time figure is deliberately not given, and the reason is itself a measurement. On the machine used for the run the baseline benchmark varied between 1.52 ms and 3.57 ms (2.3×) across identical runs, BenchmarkDotNet reported bimodal distributions on the pre-existing search benchmarks too, and the traced variant came out faster than the baseline in one run — which is impossible. A delta under that spread is not measurable there, so quoting one would be fiction.

Full details, machine configuration and the reproduce command: Benchmarks.

Bounds. A trace stops recording past Capacity (default 256) and counts the overflow in Dropped; check IsTruncated rather than assuming Steps is the whole story. Stage recording is bounded by the page or the candidate shortlist, never by the corpus, so a trace cannot grow with the index — this is what the ScoreStageIsBoundedByThePageNotTheCorpus test pins on a 20 000-doc index.

Not thread-safe. A trace is a mutable collector: give each concurrent search its own, the same way each gets its own SearchOptions. The telemetry below is the opposite case — immutable and stateless per query, so one instance is shared.

Observability (`RetrievalTelemetry`)

SearchTrace explains one search. RetrievalTelemetry answers the operational question — over time, in production, across all searches: how long they take, how many candidates they touch, how big the index is, and which stage is responsible when a query is slow.

It is opt-in and dependency-free. Attach a telemetry to an engine and it reports to a RetrievalLog sink, an IRetrievalMetrics collector, or both:

using LexiSharp.Core;

var metrics = new InMemoryRetrievalMetrics();

var engine = new RankedTextSearchEngine(
    index, new Bm25Scorer(),
    telemetry: new RetrievalTelemetry(metrics: metrics));

engine.Search("refresh token");

foreach (var e in metrics.Snapshot().Engines)
    Console.WriteLine($"{e.Engine}: {e.SearchCount} searches, {e.MinElapsedMs:0.###}–{e.MaxElapsedMs:0.###} ms");

With LexiSharp.AspNetCore, an ILogger sink is one call away:

using LexiSharp.AspNetCore;

var telemetry = RetrievalLogger.Telemetry(
    loggerFactory.CreateLogger("search"), RetrievalLogLevel.Information);

Events carry the engine name, a stable event id (search / stage / index), a self-contained message and structured ElapsedMs / Count fields, so a log aggregator can build dashboards without parsing prose. Per-stage events are Debug, so the default Information threshold keeps one line per query and drops the per-stage detail.

Instruments by pipeline stage. HybridTextSearchEngine reports one source stage per delegate (labelled by your sourceNames, so a slow lane is named the way your application named it) and one merge stage. RerankedTextSearchEngine reports retrieve and rerank:<name> — the cross-encoder call is usually where the latency lives, and it warns above 100 ms. BoostedTextSearchEngine reports retrieve and boost. A query against an empty index, or one whose terms match nothing in the vocabulary, is a Warning rather than a silent zero-result page.

Cost when unused. RetrievalTelemetry.None reports IsEnabled == false, and every instrumented engine reads that once per search before touching a timer, so an un-instrumented engine takes no timestamp, allocates nothing and calls nothing. A RetrievalTelemetry is immutable and holds no per-query state, so concurrent searches share one safely as long as the sinks are. Note this is a statement about the code, not a measurement: no timing figure is quoted here for the same reason the trace section above quotes none.

What is not covered. No exporter, aggregator or dashboard ships with any backend. The seam is IRetrievalMetrics, and RetrievalDiagnostics below turns it into a Meter and an ActivitySource that a host’s OpenTelemetry SDK — or a plain MeterListener/ActivityListener — consumes; the core package still references no package for it. InMemoryRetrievalMetrics keeps a count, a total, a min and a max per engine and stage, which is enough to assert on in tests and to back a diagnostics endpoint, and is deliberately not a percentile histogram. SearchTrace.StageCounts() reports how many steps each stage recorded; per-stage timings come from the metrics collector, not the trace.

A LexiSharpIndexHealthCheck reports readiness for the same reason: an index holding nothing answers every query with zero results, which is a silent failure rather than an obvious one.

services.AddLexiSharpRetrievalMetrics();
services.AddLexiSharpSearchHealthCheck(
    _ => index,
    o => o.Tags = ["ready"],
           o.Probe = ct => connection.OpenAsync(ct));  // optional: backing-store reachability

In-box instruments and spans (`RetrievalDiagnostics`)

The IRetrievalMetrics seam can be implemented over your own backend, and it can also be handed to RetrievalDiagnostics, which implements it over System.Diagnostics.Metrics.Meter and adds an ActivitySource. Everything it uses is in-box on .NET 10, so the core package still has no package reference: a host that already runs the OpenTelemetry SDK — or any other collector — attaches the exporter on its side.

using LexiSharp.Core;
using LexiSharp.Diagnostics;
using LexiSharp.Ranking;

using var diagnostics = new RetrievalDiagnostics();   // once per process

var engine = new RankedTextSearchEngine(
    index, new Bm25Scorer(),
    telemetry: RetrievalTelemetry.Measuring(diagnostics));

using var search = diagnostics.StartSearch(RankedTextSearchEngine.EngineName);
using (diagnostics.StartStage(RankedTextSearchEngine.EngineName, "score"))
{
    var hits = engine.Search("refresh token");
}

The meter is named LexiSharp and the source LexiSharp.Search. A MeterListener — or the OpenTelemetry SDK’s meter provider — filtered on that name sees:

instrument	kind	unit	recorded from
`search.duration`	histogram	`s`	`RecordSearch`
`search.count`	counter	`{search}`	`RecordSearch`
`search.results`	histogram	`{result}`	`RecordSearch`
`search.stage.duration`	histogram	`s`	`RecordStage`
`search.stage.count`	counter	`{invocation}`	`RecordStage`
`search.stage.items`	histogram	`{item}`	`RecordStage`
`index.documents`	gauge	`{document}`	`RecordIndex`, last value
`index.vocabulary`	gauge	`{term}`	`RecordIndex`, last value

Durations are in seconds, the unit the OpenTelemetry semantic conventions use for duration instruments. No semantic convention defines a search instrument, so the names are this library’s own; an application that serves search over HTTP keeps the HTTP layer’s http.server.request.duration for the request and reads these for the retrieval work inside it.

Listening without adding any package:

using System.Diagnostics.Metrics;

using var listener = new MeterListener();
listener.InstrumentPublished = (instrument, l) =>
{
    if (instrument.Meter.Name == RetrievalDiagnostics.MeterName)
        l.EnableMeasurementEvents(instrument);
};
listener.SetMeasurementEventCallback<double>((instrument, value, tags, _) =>
    Console.WriteLine($"{instrument.Name} {value:0.###} {instrument.Unit}"));
listener.Start();

Cardinality is part of the contract. The only tags the adapter adds are search.engine and search.stage, and both values are caller-supplied. Keep them a fixed, bounded set: one engine name per engine instance you construct, and one stage label per pipeline position (retrieve, merge, rerank:<name>, an entry of your sourceNames). Query text, result ids, document ids and user ids are never tags — every distinct value becomes a series in the backend, so a per-request value there is a slow leak. They belong in a log line or on a span you open yourself.

Spans. StartSearch opens a search span and StartStage opens a search.stage span tagged with search.stage; called inside the search scope, the stage span is its child through Activity.Current, so a trace lines up with the stages SearchTrace lists. Both return null when no ActivityListener is attached — the scopes are free in an uninstrumented host — and neither throws. The metrics callbacks cannot open the spans themselves: the seam reports after the work and carries no start timestamp or correlation id.

What it does not do. There is no exporter, no aggregation and no dashboard here: an OpenTelemetry SDK, a MeterListener or a tracing backend attaches to the names above and owns that. A gauge only knows a size that was reported while a listener had it enabled, so a listener attached after the last index change sees that engine at the next change rather than at collection time.

Per-query cost sheet (`SearchCosts`)

A trace explains one search; a telemetry aggregates every search. SearchCosts is the third thing: one request’s cost, handed back to the caller — the tokens the query was parsed into, and per stage what it processed and how long it took.

using LexiSharp.Core;

var costs = new SearchCosts();
var hits = engine.Search("refresh token", new SearchOptions(Limit: 5, Costs: costs));

Console.WriteLine($"{costs.Tokens} token(s)");

foreach (var stage in costs.Stages)
    Console.WriteLine($"{stage.Stage,-16} {stage.ItemCount,6} item(s), {stage.Windows} window(s)");

It exists for the question the other two cannot answer: how do two configurations compare, on the same queries, one at a time? An aggregate metric answers that only in the mean, and a mean hides the query where a change bought a rank and the one where it cost one. A sheet is per request and comes back with the results, so holding two side by side is a two-column table.

What the built-in engines record.

engine	rows
`RankedTextSearchEngine`	`score`
`RerankedTextSearchEngine`	`score` (its inner engine), `retrieve`, `rerank:<name>`

The inner search is asked for the full candidate depth so the reranker has room to re-order, which is why a two-stage search reports three rows rather than two — and why retrieve and score carry different item counts.

ItemCount is work, not hits. For a scoring stage it counts every document whose score was computed, including the ones that came back zero. It is deliberately not the candidateCount a telemetry reports: that one counts the matched set, and a cost sheet built on it would under-report the work by every non-matching candidate.

Windows is the column no library code fills. A scorer is a pure function of (document, query, index) — ITextScorer.Score is handed no request and cannot reach the sheet — and neither can an IReranker from Rerank. Stages that know how many windows they visited are engines, decorators and caller-side stages, which do receive the options and can Record a row themselves. A non-zero Windows therefore reports work an application measured and handed in.

Bounds, thread-safety, reuse. The stage rows stop at Capacity (default 64) and the overflow is counted in Dropped; check IsTruncated. A sheet is a mutable collector, so give each concurrent search its own, and create one per request — an instance reused across requests keeps filling.

Cost when unused. SearchOptions.Costs defaults to null and an engine reads it once per search: with no sheet attached nothing is timestamped, built or allocated, and attaching one changes nothing about what a search returns. No allocation or timing figure is quoted here because none has been measured; the trace section above gives the reason not to invent one.

Source agreement (`RetrievalAgreementAnalyzer`)

A fused ranking hides why a document is on the page. RetrievalAgreementAnalyzer reads the per-source scores in DetailedSearchResult.Contributions and classifies each document by how much its sources agree:

using LexiSharp.Core;

var page = hybrid.SearchWithDetails("refresh token", new SearchOptions(Limit: 20));
var reports = RetrievalAgreementAnalyzer.Analyze(page);

foreach (var report in reports)
    Console.WriteLine($"{report.DocumentId}  {report.Agreement}  strong: {string.Join(",", report.StrongSources)}");

var counts = RetrievalAgreementAnalyzer.Summarize(reports);

Category	What it means
`Unanimous`	several sources returned it, all found it convincing
`Disputed`	several returned it, they disagree — some convinced, some not
`Lukewarm`	several returned it, none convinced: it is on the page through a merger, not anyone’s conviction
`SingleSource`	exactly one source returned it — the case worth surfacing, since a reader of the final score cannot tell
`None`	no source returned it; should not occur on a result page

The categories are structural, not named, because source labels are yours: a setup calling them lexical/dense and one calling them bm25/cosine describe the same shape, so a hard-coded LexicalOnly would be wrong for half of them. StrongSources gives the names back.

Normalization is not optional. Raw contributions are not comparable across sources — LexiSharp’s dense lane returns a cosine in [0, 1] while its BM25 lane returns values around 1 to 10. Dividing each contribution by the best score that source gave on the same page makes the values scale-free without assuming anything about the scoring function. A source that returned nothing is reported as AbsentSources rather than weak, because being outside a source’s depth is a different fact from being ranked low by it.

The threshold is a heuristic. strongThreshold (default 0.5) is the share of a source’s best score at which a document counts as strongly supported. It is a round number chosen for readability, not a value validated against relevance data — tune it against your own corpus, and pass sourceNames so a source that silently contributed nothing still shows up as absent.

The demo is the quickest way to see the distinction matter. Running its six sample queries over its 26-document corpus, the fused page holds 9 unanimous, 8 disputed, 7 singleSource and 2 lukewarm — the obvious matches are unanimous, the arguable ones disputed, the lukewarm ones are on the page through the merger with nobody enthusiastic, and some ride on a single lane. None does not appear, and cannot: it classifies a document no source returned, so by definition it never shows up on a result page.

Calibrated confidence (`ScoreConfidence`)

BM25 output is not a probability: a score of 4.8 means nothing on its own, and the scale differs per scorer. ScoreConfidence maps a result set’s raw scores to one confidence in [0,1] per result, so a result set can be gated by an application-level minConfidence threshold that raw lexical scores cannot express:

using LexiSharp.Ranking;

// Per result, in ranking order. WinnerMargin is the default.
IReadOnlyList<double> confidences = ScoreConfidence.Compute(results);

// The one to compare against a gate. 0 for an empty result set.
double top = ScoreConfidence.TopConfidence(results, ScoreConfidenceMethod.ZScore);

Method	What it measures	Blind to
`WinnerMargin` (default)	the gap to the next result, `1 - score_next / score_current`; the last result is `0`	the absolute quality of the match — a flat ranking of high scores still looks unsure
`ZScore`	each score’s position in the set’s own distribution, squashed through the logistic function; an all-tied set collapses to a neutral `0.5` whatever its scale	the scale itself, so two corpora are not comparable

A score of exactly 0 means not a match by engine convention, so such results — and any non-finite score — are never confident. Read the name carefully: this is a relative confidence in a result set, not a calibrated probability of relevance. The threshold is yours to choose, and nothing here was fitted against relevance judgements.

Fitted confidence and the abstention threshold (`CalibratedScoreConfidence`)

Choosing that threshold by hand means guessing what a confidence of 0.6 corresponds to in terms of being right. CalibratedScoreConfidence fits the mapping instead, on labelled (score, isCorrect) pairs, and derives the threshold from them:

using LexiSharp.Ranking;

// One (score, isCorrect) pair per historical query: the top score that query produced,
// and whether that result was the one the user wanted.
var calibrated = CalibratedScoreConfidence.Fit(pairs);

double probability = calibrated.PredictProba(results[0].Score);

// The abstain decision. Untrained, this is always false — so wiring it in ahead of a Fit
// degrades nothing.
if (calibrated.ShouldAbstain(results[0].Score))
    return fallback;

`CalibrationMethod`	Shape	Fitted as
`IsotonicRegression` (default)	non-decreasing step function, interpolated between fitted points	PAVA over the pairs
`PlattScaling`	smooth sigmoid	logistic regression on the scores

The threshold is the score maximizing Youden’s J (TPR - FPR) over the labelled pairs, converted into the probability space the calibrator reports, so ShouldAbstain compares a probability against a probability. Reading it any other way would make the abstain rate a function of the corpus’s score scale rather than of the data.

Fitted on a set of pairs that all come from one corpus at one time, this is a statement about that corpus. Re-fit when the corpus changes materially.

Driving a conditional boost from it

The two halves of “boost, but only when you are unsure” are CalibratedScoreConfidence here and BoostContext<TPayload> in docs/ranking.md, and they compose through one value: the calibrator takes a raw score, and BoostContext.TopScore is one.

using LexiSharp.Ranking;

var calibrated = CalibratedScoreConfidence.Fit(pairs);   // one fit per corpus

var engine = new BoostedTextSearchEngine<CallerContext>(
    inner,
    (context, candidate) =>
        // The abstention decision, taken once per candidate but about the whole ranking.
        calibrated.ShouldAbstain(context.TopScore) && candidate.DocumentId == "b"
            ? new ScoreBoost(Add: 50)
            : ScoreBoost.None());

ShouldAbstain reads the top score, so it answers “would the base ranking have been right?” — and a boost gated on it only applies where that answer is no. TopScore is a raw score on whatever scale the inner engine produces, which is what the calibrator was fitted on; a calibrator fitted on BM25 says nothing about a dense-similarity engine’s scores, so a pipeline that changes engines has to re-fit.

For a gate that does not need labelled data, BoostContext.TopConfidence is the scale-invariant alternative — relative to this result set rather than a probability, and derived from ScoreConfidence above. The two answer the same question at different strengths, and which one is available does not depend on whether the caller has labels.

A boost that needs the caller’s own state does the same thing against BoostContext.PayloadSupplied and Payload; see the payload-less overload’s remarks in docs/ranking.md for what happens when a pipeline drops it.

Search traces and telemetry

Ranking trace (SearchTrace)

The merge’s own arithmetic

Observability (RetrievalTelemetry)

In-box instruments and spans (RetrievalDiagnostics)

Per-query cost sheet (SearchCosts)

Source agreement (RetrievalAgreementAnalyzer)

Calibrated confidence (ScoreConfidence)

Fitted confidence and the abstention threshold (CalibratedScoreConfidence)