LexiSharp

A composable information retrieval toolkit for .NET — an in-memory inverted index, four ranking strategies, rank fusion, reranking, optional PostgreSQL backends, and model-agnostic seams for dense, learned-sparse and neural scoring. The models stay in your application; the core package references no NuGet package at all.

dotnet add package LexiSharp

using LexiSharp.Core;
using LexiSharp.Indexing;
using LexiSharp.Ranking;

ITextSearchEngine engine = new RankedTextSearchEngine(
    new InMemoryTextIndex(),
    new Bm25Scorer());

engine.Index(new[]
{
    new SearchDocument("1", "The search engine uses BM25 to rank the results"),
    new SearchDocument("2", "TF-IDF is a classic method of textual search"),
    new SearchDocument("3", "Italian cuisine is renowned in Rome"),
});

foreach (var result in engine.Search("textual search"))
    Console.WriteLine($"{result.DocumentId} - {result.Score:0.###}: {result.Document.Text}");

net10.0, MIT. Optional packages: LexiSharp.MessagePack (binary index persistence), LexiSharp.AspNetCore (a GET /search endpoint), LexiSharp.Postgres (lexical, vector, sparse, fuzzy and true BM25 backends). The engine is also a .NET tool — dotnet tool install -g LexiSharp.Cli, then lexisharp search ./docs --query "rank fusion".

See it running

dotnet run --project samples/LexiSharp.Demo
# → http://localhost:5000

Five retrieval strategies over one corpus, compared live — plain BM25, corpus-derived semantic expansion, dense hashing embeddings, reciprocal-rank fusion and a term-overlap rerank — with per-lane latency, highlighting and a click-through “why did this rank here?” panel. No model, no external service.

Clicking a hit also shows the SearchTrace chain for that document — the stages it passed through and the score going in and out of each:

score   4.6925 -> 4.6925  BM25
score   3.1642 -> 3.1642  BM25
score   0.4867 -> 0.4867  Dense
merge   0.0492 -> 0.0492  dense=0.4867 lexical=4.6925 semantic=3.1642
rerank  0.0492 -> 100     CrossEncoder(QueryTermOverlap)

Each lane states what backs it — the semantic lane uses PmiTermExpander, the dense lane a HashingEmbeddingProvider, the rerank lane a local term-overlap ICrossEncoderScorer — and all three are swappable for a real model behind their existing seam. See Reference demo for the wiring.

Want evidence rather than a demo? The same engines are scored on NFCorpus, SciFact and ArguAna against BEIR’s published BM25 numbers in the eval harness, which you can run yourself.

What is measured, and what is not

Stated up front, because a library that only quotes its good numbers is not worth reading twice. Every figure below is reproducible with a command on the Evaluation page.

• Against published baselines. On three public BEIR corpora, with the analysis, the BM25 parameters and the metric convention aligned to those the reference figures were produced with, the plain BM25 scorer reaches nDCG@10 0.3218 on NFCorpus and 0.6789 on SciFact, against 0.3218 / 0.6789 — equal to the fourth decimal, with an index that matches the reference’s term for term. On ArguAna, at the reference’s own k1=0.9/b=0.4 and under the same conventions, it reaches 0.4005 against 0.3970, 0.0035 short. Under the library’s own defaults the same scorer reads 0.308 / 0.662 / 0.320; that difference is the configuration, not the ranking. The corpora are md5-verified on download, the harness ships no licence over them, and every number here is pinned and re-checked by the Pinned reference workflow.
• A claim withdrawn before it was published. An earlier figure on ArguAna is not reproducible by any code path in this repository, and no release carried it; the measurement behind the withdrawal is in evaluation, the provenance in the changelog.
• Where the composed stack pays off. Fusing a dense retriever with BM25 beats both parents on all three corpora: NFCorpus 0.333 against 0.308/0.304, SciFact 0.686 against 0.662/0.655, ArguAna 0.350 against 0.320/0.313. A cross-encoder on top is corpus-dependent: it improves every first stage on NFCorpus (0.346, about 2.1 points above BEIR’s BM25), helps one of three on SciFact, and loses to all three on ArguAna. Those two lanes are the only ones that use a real model, and the model comes from the harness, not from the library.
• The SQL backends, on the same corpora. The PostgreSQL text, BM25 and vector engines are scored over the same queries and qrels as the in-memory table: at matched k1=0.9/b=0.4, ParadeDB’s Tantivy BM25 reads 0.3055 and 0.6633 on NFCorpus and SciFact against this library’s 0.305 and 0.655, ts_rank_cd reads 0.1864 and 0.0188 — mostly its matcher, which ANDs every term — and pgvector at ef_search 10/40/160 reads 0.2882 / 0.2960 / 0.3013 on NFCorpus against the exact in-memory dense row’s 0.304, and 0.5998 / 0.6528 / 0.6550 on SciFact against 0.655: that gap is the index’s cost in nDCG. ArguAna produces no row from any of the three: the text engines reject 2 of its 1,406 test queries, and the vector engine rejects 148 as phrases. None of these rows is pinned, and the text engines’ analysis is PostgreSQL’s rather than this library’s. The sparse engine, like the dense one, is measured in memory and through the SQL backend on the same vectors: on ArguAna under --exclude-query-doc, SPLADE-v3 reads 0.5216 in memory and 0.5200 through pgvector’s HNSW index, so 0.0002 of that difference is the index and the rest is the same dot product computed in SQL. See evaluation.
• The fuzzy engine is measured, and it is not a prose ranker. pg_trgm similarity reads 0.0000 on NFCorpus and 0.0033 on SciFact at its default 0.3 threshold, and 0.0981 / 0.2186 at the low end of an in-sample threshold sweep — against BM25’s 0.308 / 0.662. The engine matches labels and typos; the rows are what a threshold on character trigrams costs on prose.
• Zero runtime dependencies in the core. LexiSharp references no NuGet package at all: index, BM25, every scorer and every decorator are BCL only. The optional packages bring their own.
• Behaviour is specified by tests. Everything documented here is covered by the xUnit suite; the Postgres and ParadeDB integration tests run against a live instance on every build and self-skip when POSTGRES_TEST_CONNECTION is unset.
• Not measured, and named rather than glossed over. The time cost of SearchTrace (its allocation is pinned; the timing was not resolvable on the machine used, so no figure is quoted). The learned-sparse lane runs no model: it scores vectors a SPLADE export produced offline, and its rows are that model’s, not this library’s. Corpus-derived expansion is a loss where it has been measured (0.8469 against BM25’s 0.8751 on the reference corpus). Not every combination of engine, scorer, reranker and merger is exercised by a test.

Where to go next

Page	What is in it
Getting started	the engine in three lines, the typed LexiSharpIndex<T> facade, document loaders, an ASP.NET Core endpoint, the demo app
Building an inverted index	the inverted index and its statistics, named text fields, the tokenizer and stemming, binary persistence
Querying: filters, facets, highlighting	metadata filters, pagination, phrase queries, highlighting, prefix & fuzzy terms, synonyms, facets, the span-first API
BM25 and ranking functions	four scorers, BM25+ / BM25L and their δ tuners, BM25F, score boosting, proximity, per-term explanations — each with the measurement behind it
Rerankers and hybrid search	rerankers (MMR, cascade, cross-encoder, MaxSim), hybrid federation, cost-based and intent-based routing
Dense, sparse and query expansion	dense, learned-sparse and corpus-derived expansion, all behind consumer-supplied seams
Classification and text analysis	lexical similarity, keyword extraction, Naive Bayes, trees/forest/linear, centroids, nearest neighbours and blends
PostgreSQL and ParadeDB backends	PostgreSQL lexical / vector / sparse / fuzzy, ParadeDB BM25, what the integration tests cover
Search traces and telemetry	SearchTrace, production telemetry, source agreement, calibrated score confidence
Retrieval quality and IR metrics	the BEIR numbers in full, IR metrics, the benchmark CLI, per-query diffs, parameter tuning
Allocation and latency benchmarks	allocations and timings, with the machine they came from and the command to reproduce them
Reference	packages, architecture, scoring conventions, scope and limits, build and test
SPLADE: learned sparse retrieval	writing a consumer-side ISparseEmbeddingProvider around a SPLADE ONNX model
API reference	every public type of the four packages, generated from the assemblies — the contracts you implement, and the classes that implement them

Scope

Version 0.9.0, and the public API may still change between minor versions — pin a version and read the release notes. The changelog records what changed in each release. The techniques implemented (BM25 and its variants, RRF, SPLADE-style sparse retrieval, MaxSim) follow published information-retrieval literature; what is here is a small, dependency-free .NET implementation of them, not new research. Full scope and limits.

All projects · Source on GitHub · NuGet · MIT