Getting started

Four ways in, from smallest to largest: the three-line engine, the typed facade, loaders for data on disk, and an HTTP endpoint. The home page has the shortest version of all, and the demo (See it running) is worth running once before reading any of this.

Everything on this page is in the core package unless a section says otherwise: the namespaces LexiSharp.Sources, LexiSharp.Classification and the rest all ship inside LexiSharp. The separate packages are only LexiSharp.MessagePack, LexiSharp.AspNetCore and LexiSharp.Postgres.

Quick start 

using LexiSharp.Core;
using LexiSharp.Indexing;
using LexiSharp.Ranking;

ITextSearchEngine engine = new RankedTextSearchEngine(
    new InMemoryTextIndex(),
    new Bm25Scorer());

engine.Index(new[]
{
    new SearchDocument("1", "The search engine uses BM25 to rank the results"),
    new SearchDocument("2", "TF-IDF is a classic method of textual search"),
    new SearchDocument("3", "Italian cuisine is renowned in Rome"),
});

IReadOnlyList<SearchResult> results = engine.Search("textual search");

foreach (var result in results)
    Console.WriteLine($"{result.DocumentId} - {result.Score:0.###}: {result.Document.Text}");

Drop-in index (LexiSharpIndex<T>)

The fastest way in: a typed facade that maps your own documents to the engine and back, so you go from “a list of objects” to “working search” in a few lines. TDocument can be string or SearchDocument (selectors default to the obvious mapping), or any class with explicit id/text selectors:

using LexiSharp;

var search = new LexiSharpIndex<MyDocument>(o =>
{
    o.Id = d => d.Id;
    o.Text = d => d.Body;
    o.EnableFuzzy = true;        // plain terms behave like `term~1`
});

search.Add(documents);

var hits = search.Search("architecture distributed systems");
foreach (var hit in hits)
    Console.WriteLine($"{hit.DocumentId} - {hit.Score:0.###}: {hit.Document.Body}");

Search returns typed hits carrying the original document; SearchWithFacets adds facet buckets over the match set, Explain returns the per-term score breakdown, Statistics a corpus snapshot. Highlight: true on the query options wraps the matched terms of each hit. The scorer/tokenizer knobs of the rest of the library stay reachable through LexiSharpIndexOptions<T> (UseBm25, UseTfIdf, UseQueryLikelihood, UseBoolean, RemoveStopWords, Stemmer, NGramMax, Synonyms, Reranker, …).

From the terminal (LexiSharp.Cli)

The same engine is a .NET tool, for trying it on a directory before writing any code:

dotnet tool install -g LexiSharp.Cli

lexisharp search ./docs --query "rank fusion"          # scan and search, writes nothing
lexisharp segment build ./docs                         # index a store at ./.lexisharp
lexisharp search --query "rank fusion"                 # finds it: no directory, no store flag
lexisharp segment info .lexisharp

A directory that holds a store is searched through the store rather than re-scanned: the directory itself if it is a store, else its .lexisharp. benchmark, diff, baseline and verify run a corpus against labelled queries and report the IR metrics of the evaluation page; lexisharp benchmark --help lists the configurations and the file formats. The tool’s own README is here.

Document sources (LexiSharp.Sources)

Pair the facade with the loaders to index data that lives on disk: each loader produces a LoadedDocument (id + text + optional fields/category) that maps straight into a LexiSharpIndex<LoadedDocument> or a SearchDocument.

using LexiSharp.Sources;

var notes = MarkdownLoader.LoadDirectory(@"./notes");          // *.md, *.markdown, *.mdx
var ledgers = TextFileLoader.ScanDirectory(@"./ledgers");      // any plain text files
var records = JsonDocumentsLoader.Parse(json, new JsonDocumentLoadOptions
{
    IdProperty = "docid",
    TextProperty = "content",
    CategoryProperty = "bucket",
});
  • MarkdownLoader — reads a file (LoadFile) or a whole directory (LoadDirectory, recursive, hidden paths skipped), parses the optional --- YAML front matter into fields (title, category, tags as [a, b] lists, plus any other key: value), and indexes the remaining text. Document ids default to the full path for files and the forward-slash relative path for directory scans.
  • TextFileLoader — whole file content is the indexed text; title and source fields are added automatically.
  • JsonDocumentsLoader — parses a JSON array of objects; scalars become fields, scalar arrays are flattened to a comma-separated string; the id/text/category property names are configurable.

ASP.NET Core search endpoint (LexiSharp.AspNetCore)

A minimal-API endpoint that exposes any registered LexiSharpIndex<T> over HTTP — a thin package built on the core (no web framework of its own; you reference it from your ASP.NET Core app):

using LexiSharp;
using LexiSharp.AspNetCore;

var builder = WebApplication.CreateBuilder(args);

var index = new LexiSharpIndex<SearchDocument>();
index.AddRange(loader.LoadDirectory("data"));
builder.Services.AddSingleton(index); // resolve the same<T> the endpoint uses

var app = builder.Build();
app.MapLexiSharpSearch<SearchDocument>(); // GET /search?q=...&limit=10&offset=0&highlight=true

app.Run();
  • Query parameters: q (required), limit (default 10, at most 100), offset (default 0, at most 10 000), minimumScore (default −∞), highlight (default false).
  • A blank q returns 400 with a { "field": "q", "message": ... } error payload. So does a request the search refuses — a page past the bounds above, or a query the engine will not run (more than 32 distinct term*/term~ atoms, syntax this index cannot honor) — because each of those is about the request and retrying it cannot work.
  • With highlight=true each hit carries highlightedText, the document text with matched terms wrapped in <em>; the page honors Limit/MinimumScore/Offset exactly like the in-process LexiSharpIndex<T>.Search. The document’s own characters arrive HTML-encoded (&lt;, &amp;, &quot;), so the value can go into an element’s content as it is; the tags around a match are the only tags in it, and the route offers no way to turn that off.

Exposing the route

The endpoint is a convenience surface, and the bounds above are what make one request cost a bounded amount of work. Three things stay the application’s:

  • Authorization. The route carries none. app.MapLexiSharpSearch<T>() returns an IEndpointConventionBuilder, so RequireAuthorization() and the rest of the endpoint conventions apply: app.MapLexiSharpSearch<SearchDocument>().RequireAuthorization();
  • Rate limiting. Nothing throttles repeated requests; use ASP.NET Core’s rate limiter on the route when the corpus is not public.
  • What a hit carries. LexiSharpSearchHit<TDocument>.Document is the caller’s own object, serialized whole. A corpus whose documents hold fields a caller should not read needs a projection type (index a document type that carries only what may be published) or a hand-written endpoint; the shipped one does not select fields.

The route passes HttpContext.RequestAborted into the search, so a client that disconnects stops the ranking it started. LexiSharpSearchEndpoint.Handle has the same overload for a host that maps the route itself.

Change the ranking algorithm without rebuilding the index:

var bm25Engine   = new RankedTextSearchEngine(index, new Bm25Scorer());
var tfIdfEngine  = new RankedTextSearchEngine(index, new TfIdfScorer());
var booleanEngine = new RankedTextSearchEngine(index, new BooleanScorer(BooleanMatch.AllTerms));

Reference demo (samples/LexiSharp.Demo)

A self-contained ASP.NET Core app that runs five retrieval strategies over the same corpus and compares them live — plain BM25, corpus-derived semantic expansion, dense hashing embeddings, reciprocal-rank fusion, and a term-overlap rerank — with per-lane latency and highlighting. Clicking any hit opens a “why did this rank here?” panel powered by LexiSharpIndex.Explain (per-term contributions, IDF, length) or, for the federated/dense lanes, the per-source scores from HybridTextSearchEngine.SearchWithDetails. No external model or dependency: the semantic lane uses PmiTermExpander, the dense lane a HashingEmbeddingProvider, and the rerank lane a local term-overlap ICrossEncoderScorer — swap any of them behind its seam for a real model.

dotnet run --project samples/LexiSharp.Demo
# → http://localhost:5000  (search box + five comparison columns)

See See it running for what this buys you and how to read the lanes.

Demo corpora and segmentation

The corpus is the built-in one, written for the demo. --corpus runs the same five lanes over a BEIR corpus instead — nfcorpus, scifact or arguana — read from the evaluation harness’s data directory, which the harness downloads on first use:

dotnet run --project samples/LexiSharp.Demo -- --corpus scifact

The example queries come from the corpus’s own queries.jsonl, restricted to the ones carrying a positive judgement in qrels/test.tsv. The demo builds a BEIR corpus’s documents the way the harness does — title as a text field, body as the text — but it does not rank it identically, because the two tokenize differently: the demo removes stop words and the harness’s default analysis does not. On NFCorpus, 5 of the 6 example queries come back with a different top-10 ordering, for a mean top-10 overlap of 6.67. Compare a demo figure against the harness by running the harness.

--segmentation picks how a separator inside a word is treated: uax29 (the default) or flat. Under flat every non-word character ends a word, so 1,000 indexes as the term 000 and don't as don; under uax29 a comma between two digits and an apostrophe within one class stay in the token. It decides which tokens exist, so every lane uses it — a page comparing two lanes under two segmentations would be comparing tokenizers.

This site uses Just the Docs, a documentation theme for Jekyll.