LexiSharp 0.4.0

LexiSharp

A lightweight lexical text search and classification library for .NET.

LexiSharp provides a small, dependency-free set of interfaces and implementations for indexing plain text and retrieving/ranking/classifying documents without any semantic or ML model — pure lexical statistics.

Status & scope

• Version 0.4.0, single maintainer. The library is young and its public API may still change between minor versions — pin a version and read the release notes if you adopt it early. Contributions and feedback are welcome.
• Built on established IR. The techniques implemented (BM25, RRF, SPLADE-style sparse retrieval, MaxSim) follow well-documented information-retrieval literature; the value here is a small, dependency-free .NET implementation of them, not new research.
• Behavior is specified by tests. The documented behavior below is covered by the xUnit suite; the Postgres/ParadeDB integration tests run against a live instance when POSTGRES_TEST_CONNECTION is set and self-skip otherwise (see Building & testing). Performance numbers live in BENCHMARKS.md and are indicative — always measure on your own corpus.
• Combinatorial coverage. Engines, scorers, rerankers and mergers are tested individually and in documented combinations; not every pairing is exercised yet, so treat an unusual combination as supported but unproven until you test it on your data.

Features

• Pluggable architecture: an ITextIndex, ITextScorer and ITokenizer are independent contracts; algorithms can be swapped without touching the engine.
• Drop-in entry point (LexiSharpIndex<T>): a typed facade that maps your own document type to the engine and back — index objects, get your objects back — with fluent options for the scorer, tokenizer, fuzzy matching, synonyms and an optional reranker.
• Document sources (LexiSharp.Sources): loaders for the data on your disk — markdown with YAML front matter, plain text files and JSON arrays — producing id + text + fields + category records ready to index.
• Benchmark CLI (LexiSharp.Cli + LexiSharp.Benchmarking): an in-core runner compares stock scorers, tuned BM25 and RRF hybrids on your own corpus with labeled queries, reporting the standard retrieval metrics and latency; a console front-end drives it from the command line.
• ASP.NET Core endpoint (LexiSharp.AspNetCore): a minimal-API extension that maps any LexiSharpIndex<T> to a GET /search endpoint with pagination, minimum score and text highlighting over HTTP.
• Four ranking strategies behind the same ITextSearchEngine:
  • Bm25Scorer — Okapi BM25, with ready-made Bm25Parameters profiles (Balanced, Aggressive, Conservative),
  • TfIdfScorer — TF-IDF,
  • QueryLikelihoodScorer — probabilistic language model (Jelinek-Mercer smoothing),
  • BooleanScorer — exact AND/OR filter.
• In-memory inverted index (InMemoryTextIndex) with term positions, document frequencies, corpus statistics and incremental Add/Remove, plus an index statistics snapshot (GetStatistics: documents, vocabulary, tokens, average length, vocabulary richness).
• Score boosting (BoostedTextSearchEngine): a decorator that applies signed score adjustments (multiplicative factor and/or additive offset) per result — boost a category or a priority, damp or penalize stale matches — without touching the underlying engine.
• Second-stage reranking: an IReranker seam and the RerankedTextSearchEngine decorator (over-fetch, re-rank, guard rails) in the core; shipped rerankers include a diversity-preserving MMR, a cascade pipeline that chains any number of reranking stages with per-stage trimming, a cross-encoder reranker driven by a consumer-provided pairwise scoring model (ICrossEncoderScorer), and a ColBERT MaxSim reranker that re-scores a shortlist token-by-token with late interaction (ITokenEmbeddingProvider).
• Sparse learned embeddings: SparseTextSearchEngine and its ISparseEmbeddingProvider seam bring SPLADE/uniCOIL-style retrieval (.NET-core only, weights learned, inverted-index scoring kept) without pulling ONNX into the library — the model lives in the consumer.
• Semantic lexical expansion (LexiSharp.Expansion): an ITermExpander seam that widens a query — and any document, at index time — with corpus-derived related terms (PmiTermExpander learns PPMI/co-occurrence associations from your own documents), then folds them into the classic inverted index (ExpansionTextIndex, zero-new-dependencies, pure .NET core; the seam is where a real neural SPLADE model plugs in later).
• Reference demo app (samples/LexiSharp.Demo): an ASP.NET Core page that compares BM25, semantic expansion, hybrid RRF and cross-encoder rerank side by side on one corpus, with latency, highlighting and a click-through "why did this rank here?" explanation — no model, no external service.
• Metadata filters: declarative, AND-composed filters over document fields (MetadataFilterOperator: equal, not-equal, contains, numeric-or-ordinal greater/less than) in SearchOptions — honored by every backend (stock in-memory and SQL) before scoring.
• Pagination: SearchOptions.Offset cuts any window [Offset, Offset + Limit) of the ranking — honored by the stock engine, the boost/rerank decorators, the hybrid merger (the page comes from the merged ordering) and every SQL backend.
• Phrase queries: double-quoted segments ("machine learning") must appear at consecutive document positions (several phrases are AND-ed), while the free terms around them keep scoring — free terms never hard-filter a mixed query. Natively honored by the stock engine (index positions), PostgreSQL (websearch_to_tsquery) and ParadeDB (###).
• Highlighting (LexiSharp.Highlighting): TextHighlighter wraps query matches in the original text (HighlightFull) or returns padded, word-snapped snippets (Highlight) — driven by the tokenizer's span mode (TokenizeWithSpans: term + [Start, Length) offsets into the source).
• Prefix & fuzzy queries: neural* expands to every indexed term with that prefix, catt~/catt~N fuzzy-matches within N edits (default 1, clamped to 0–2) — search-time vocabulary expansion on the stock engine (IVocabularyIndex), 64 terms max per atom. A selective mode (FuzzyOnlyOutOfVocabulary, or SearchOptions of the same name) keeps a term already in the vocabulary exact, so only genuinely unknown words are corrected and correctly spelled tokens can no longer be degraded by close variants.
• Synonyms (SynonymMap): one-way rewrites and bidirectional equivalence groups, tokenized at engine construction and applied to free query terms — one level deep (non-transitive), never inside quoted phrases.
• Facets (IFacetedSearchEngine): SearchWithFacets returns the ranked page plus value counts per requested Fields entry over the whole match set — independent of Offset/Limit, ordered by count then value.
• Span-first API: the text entry points — ITokenizer.Tokenize/TokenizeWithSpans, QueryParser.Parse/SplitRaw, ITextSearchEngine.Search, IFacetedSearchEngine.SearchWithFacets and RankedTextSearchEngine.Explain — each have a ReadOnlySpan<char> overload that avoids materializing the query as a string; default interface implementations forward to the string path so existing implementers keep working.
• Cost-based routing (RoutedSearchEngine): opt-in decorator over several pre-filled engines that forwards each query to the cheapest one, chosen by an IQueryCostEstimator — the default CheapestByCandidateCountEstimator uses per-engine IQueryCostProbe estimates.
• Intent-based routing (RoutingSearchEngine): opt-in decorator over pre-composed routes (engine + optional metadata filters) that asks an IQueryRouter you supply — a rule, a classifier or a model — which route to run, with a confidence threshold and a fallback. The route's filters are AND-ed onto the caller's. The seam is model-agnostic: LexiSharp never runs a model itself.
• Lexical similarity (LexiSharp.Similarity): pairwise token-set measures (Jaccard, Sørensen–Dice) over the library tokenizer, a pg_trgm-style character trigram similarity, and a rolling Levenshtein edit distance — near-duplicate detection and fuzzy matching with no index.
• Keyword extraction (LexiSharp.Keywords): a corpus-backed TF-IDF extractor (demotes corpus-frequent words) and a graph-based TextRank extractor (weighted co-occurrence graph + PageRank), both deterministic and tokenizer-configurable.
• Explainable scoring: Bm25Scorer, TfIdfScorer and QueryLikelihoodScorer implement IScoreExplainer, and RankedTextSearchEngine.Explain returns a per-term breakdown (TF, IDF, term score, length normalization, parameter values) of any ranking decision.
• Calibrated confidence (ScoreConfidence): maps a result set's raw scores — BM25 output and friends, whose scale is not a probability — to a per-result confidence in [0,1], either from the winner margin (gap to the next result, scale-invariant) or from a logistic z-score against the set's own distribution. TopConfidence feeds a minConfidence gate that raw lexical scores cannot.
• Evaluation metrics (RetrievalMetrics): Precision@k, Recall@k, F1@k, binary and graded nDCG@k (exponential gains), plus ReciprocalRank@k (→ MRR) and AveragePrecision@k (→ MAP).
• BM25 tuning: Bm25ParameterTuner grid-searches k1/b against your own validation queries, judged by Precision@k, Recall@k, F1@k or nDCG@k.
• Supervised classification (NaiveBayesClassifier): multinomial Naive Bayes with Laplace smoothing, exposing a dedicated ITextClassifier interface.
• Configurable tokenizer: Unicode NFKD normalization and diacritics removal, lowercasing, optional stop-word removal, optional n-grams, and a pluggable IStemmer seam (no stemmer ships with the library — bring your own, e.g. Snowball). Tokenization is SIMD-accelerated (SearchValues + IndexOfAnyExcept, with a System.Text.Ascii fast path in normalization).
• Optional backends, shipped as separate packages:
  • LexiSharp.Postgres — PostgreSQL backends implementing the same ITextSearchEngine: a lexical engine over tsvector + GIN + unaccent, an ANN engine over pgvector (HNSW/IVFFlat) driven by an external IEmbeddingProvider, a learned-sparse engine over pgvector sparsevec (HNSW) driven by an ISparseEmbeddingProvider, an approximate fuzzy engine over the pg_trgm trigram extension (with optional fuzzystrmatch refinement), and a true Okapi BM25 engine over the ParadeDB pg_search Tantivy extension (ParadeDBTextSearchEngine, AGPL-3). The extension engines share one documents table and are picked at instantiation, exactly like the other backends.

Quick start

using LexiSharp.Core;
using LexiSharp.Indexing;
using LexiSharp.Ranking;

ITextSearchEngine engine = new RankedTextSearchEngine(
    new InMemoryTextIndex(),
    new Bm25Scorer());

engine.Index(new[]
{
    new SearchDocument("1", "The search engine uses BM25 to rank the results"),
    new SearchDocument("2", "TF-IDF is a classic method of textual search"),
    new SearchDocument("3", "Italian cuisine is renowned in Rome"),
});

IReadOnlyList<SearchResult> results = engine.Search("textual search");

foreach (var result in results)
    Console.WriteLine($"{result.DocumentId} - {result.Score:0.###}: {result.Document.Text}");

Drop-in index (LexiSharpIndex<T>)

The fastest way in: a typed facade that maps your own documents to the engine and back, so you go from "a list of objects" to "working search" in a few lines. TDocument can be string or SearchDocument (selectors default to the obvious mapping), or any class with explicit id/text selectors:

using LexiSharp;

var search = new LexiSharpIndex<MyDocument>(o =>
{
    o.Id = d => d.Id;
    o.Text = d => d.Body;
    o.EnableFuzzy = true;        // plain terms behave like `term~1`
});

search.Add(documents);

var hits = search.Search("architecture distributed systems");
foreach (var hit in hits)
    Console.WriteLine($"{hit.DocumentId} - {hit.Score:0.###}: {hit.Document.Body}");

Search returns typed hits carrying the original document; SearchWithFacets adds facet buckets over the match set, Explain returns the per-term score breakdown, Statistics a corpus snapshot. Highlight: true on the query options wraps the matched terms of each hit. The scorer/tokenizer knobs of the rest of the library stay reachable through LexiSharpIndexOptions<T> (UseBm25, UseTfIdf, UseQueryLikelihood, UseBoolean, RemoveStopWords, Stemmer, NGramMax, Synonyms, Reranker, ...).

Document sources (LexiSharp.Sources)

Pair the facade with the loaders to index data that lives on disk: each loader produces a LoadedDocument (id + text + optional fields/category) that maps straight into a LexiSharpIndex<LoadedDocument> or a SearchDocument.

using LexiSharp.Sources;

var notes = MarkdownLoader.LoadDirectory(@"./notes");          // *.md, *.markdown, *.mdx
var ledgers = TextFileLoader.ScanDirectory(@"./ledgers");      // any plain text files
var records = JsonDocumentsLoader.Parse(json, new JsonDocumentLoadOptions
{
    IdProperty = "docid",
    TextProperty = "content",
    CategoryProperty = "bucket",
});

• MarkdownLoader — reads a file (LoadFile) or a whole directory (LoadDirectory, recursive, hidden paths skipped), parses the optional --- YAML front matter into fields (title, category, tags as [a, b] lists, plus any other key: value), and indexes the remaining text. Document ids default to the full path for files and the forward-slash relative path for directory scans.
• TextFileLoader — whole file content is the indexed text; title and source fields are added automatically.
• JsonDocumentsLoader — parses a JSON array of objects; scalars become fields, scalar arrays are flattened to a comma-separated string; the id/text/category property names are configurable.

Benchmark CLI (LexiSharp.Cli)

Compare ranking strategies over your own corpus without writing code. The runner (LexiSharp.Benchmarking in the core) builds one shared in-memory index, evaluates every selected configuration against the same labeled queries and reports nDCG/MAP/MRR/Recall/ Precision/F1 at the retrieval depth plus the per-query latency:

dotnet run --project bench/LexiSharp.Cli -c Release -- benchmark ./notes \
    --queries queries.json --qrels qrels.tsv --top-k 10

• --queries accepts a JSON object { "id": "query text", ... } or a TSV id⇥text.
• --qrels is a TSV qid⇥docid[⇥grade] (grades are read as binary relevance); a query without any judgment is loaded but excluded from the metric averages.
• --configs selects the comparison: bm25, bm25-tuned (fits (k1, b) on the labeled queries), tfidf, ql (query likelihood), hybrid (RRF over BM25 + TF-IDF).
• --json <path> writes the results as a machine-readable report.

The same comparison is available in-process through CorpusBenchmark.Run over any IReadOnlyCollection<SearchDocument> and BenchmarkQuery set, with custom engines reachable through the public BenchmarkConfig constructor.

ASP.NET Core search endpoint (LexiSharp.AspNetCore)

A minimal-API endpoint that exposes any registered LexiSharpIndex<T> over HTTP — a thin package built on the core (no web framework of its own; you reference it from your ASP.NET Core app):

using LexiSharp;
using LexiSharp.AspNetCore;

var builder = WebApplication.CreateBuilder(args);

var index = new LexiSharpIndex<SearchDocument>();
index.AddRange(loader.LoadDirectory("data"));
builder.Services.AddSingleton(index); // resolve the same<T> the endpoint uses

var app = builder.Build();
app.MapLexiSharpSearch<SearchDocument>(); // GET /search?q=...&limit=10&offset=0&highlight=true

app.Run();

• Query parameters: q (required), limit (default 10), offset (default 0), minimumScore (default −∞), highlight (default false).
• A blank q returns 400 with a { "field": "q", "message": ... } error payload.
• With highlight=true each hit carries highlightedText, the document text with matched terms wrapped in <em>; the page honors Limit/MinimumScore/Offset exactly like the in-process LexiSharpIndex<T>.Search.

Change the ranking algorithm without rebuilding the index:

var bm25Engine   = new RankedTextSearchEngine(index, new Bm25Scorer());
var tfIdfEngine  = new RankedTextSearchEngine(index, new TfIdfScorer());
var booleanEngine = new RankedTextSearchEngine(index, new BooleanScorer(BooleanMatch.AllTerms));

Semantic lexical expansion (LexiSharp.Expansion)

A query (or a document) rarely uses the exact vocabulary of what it is about. The ITermExpander seam lets the index enrich itself: each document is indexed as usual, then gains a handful of expansion terms — related words learned from the corpus — stored at synthetic positions after its literal tokens (a phrase query can never bridge the boundary). PmiTermExpander learns those associations statistically from your own documents (windowed co-occurrence, PMI-filtered, density-capped so function words don't leak); plug your own implementation (e.g. a neural SPLADE model) behind the same interface:

using LexiSharp;
using LexiSharp.Expansion;

// Learn the associations from the documents you are about to index (say, loaders output).
var expansion = PmiTermExpander.LearnFrom(corpus, options: new PmiTermExpanderOptions
{
    ContextWindowSize = 8,     // ±7 neighbours; 8 is the standard (LSA/PMI) default
    MaxWindowDensity = 0.5,    // drop words present in > half the windows (function words)
    MaxTotalTerms = 8,         // expansion budget per document
});

var index = new LexiSharpIndex<SearchDocument>(options => options.TermExpander = expansion);
index.AddRange(documents); // every document is expanded lazily at its own Add

var hits = index.Search("refresh"); // also finds "token", "expiry", "session" documents

The rediscovered terms enter the very same inverted index — BM25, phrase, highlighting and every engine work on the enriched vocabulary untouched. The bm25-semantic preset of the benchmark CLI measures the impact against plain BM25 on your own corpus.

Reference demo (samples/LexiSharp.Demo)

A self-contained ASP.NET Core app that runs four retrieval strategies over the same corpus and compares them live — plain BM25, corpus-derived semantic expansion, reciprocal-rank fusion, and a cross-encoder rerank — with per-lane latency and highlighting. Clicking any hit opens a "why did this rank here?" panel powered by LexiSharpIndex.Explain (per-term contributions, IDF, length) or, for the federated lanes, the per-source scores from HybridTextSearchEngine.SearchWithDetails. No external model or dependency: the semantic lane uses PmiTermExpander and the rerank lane a local term-overlap ICrossEncoderScorer — swap either behind its seam for a real model.

dotnet run --project samples/LexiSharp.Demo
# → http://localhost:5000  (search box + four comparison columns)

The demo is the fastest way to see what the composable pieces buy you; its whole wiring is DemoSearchService (four engines over one corpus) plus a single static wwwroot/index.html.

Classification

using LexiSharp.Classification;

var classifier = new NaiveBayesClassifier();
classifier.Train(trainingDocuments); // requires a non-null SearchDocument.Category

foreach (var prediction in classifier.Predict("i cannot connect to the internet"))
    Console.WriteLine($"{prediction.Category}: {prediction.Probability:P}");

The classifier is a IWeightedPredictor too (classifier is IWeightedPredictor): a spell-corrected token can carry less evidence than an exact match by passing WeightedTokens directly. Predict/PredictBest accept a set of excludedCategories to hide hot categories at runtime without retraining (probabilities renormalize over the rest). NaiveBayesOptions tunes the scoring: a softmax Temperature (sharpening/flattening), an IdfMode (None / DocumentCount = log(1 + N/df) / ClassCount = max(0, log(C/df))), an Alpha smoothing coefficient (optionally applied to the priors through SmoothPriors) and SkipOutOfVocabularyTokens (ignore unknown query terms instead of a Laplace penalty). Setting Complement switches to Complement Naive Bayes (Rennie et al. 2003, matching scikit-learn's ComplementNB): each class is learned from the complement of its documents and a query is attributed to the class whose exclusion explains it least — a cheap robustness win when the training labels are heavily imbalanced. Train must not overlap any Predict; concurrent Predict calls are safe.

Tokenizer customization

using LexiSharp.Linguistics;

var tokenizer = new Tokenizer(new TokenizerOptions
{
    RemoveStopWords = true,       // English list, or provide StopWords.Create(...)
    NGramMax = 2,                 // produce unigrams + bigrams
    Stemmer = new MyStemmer(),    // implement IStemmer (French, Snowball, ...)
});

Score boosting (BoostedTextSearchEngine)

Wrap any engine to boost or damp its ranking without changing the engine. The boost is a function of the whole result, so it can read the score, the document metadata or external data (a closure over your own store):

ITextSearchEngine boosted = new BoostedTextSearchEngine(baseEngine,
    result =>
    {
        double factor = 1.0;
        if (result.Document.Category == "priority")
            factor = 2.0;                                  // up-weighted metadata
        if (result.Document.Fields.TryGetValue("stale", out _))
            factor *= 0.5;                                 // damp old matches

        return new ScoreBoost(Multiply: factor, Add: -0.5); // factor and/or offset, signed
    });

Writes are forwarded to the inner engine; Search applies score * Multiply + Add to every candidate, then re-sorts and re-applies Limit/MinimumScore. A plain double is accepted as a multiplicative factor (result => 2.0). Positive boosts (factor > 1, positive offset) and negative ones (factor in (0, 1) damp, negative offset penalty) are equally expressible; factor 0 drops the document entirely. The decorator requests more candidates than the final limit (maxCandidates, default 50) so boosted documents can surface.

Reranking (IReranker, MMR, cascade)

Retrieve with recall, then re-rank a shortlist with precision. The core seam is IReranker; RerankedTextSearchEngine decorates any engine (over-fetches, re-ranks, applies MinimumScore/Limit on the final scores):

using LexiSharp.Core;

IReranker reranker = ...;                                    // yours, or the MMR one below
ITextSearchEngine engine = new RerankedTextSearchEngine(baseEngine, reranker, maxCandidates: 100);

LexiSharp ships two built-in rerankers. MMR (Maximal Marginal Relevance) re-orders candidates so each next pick is relevant and different from the picks before it — near-duplicate results are pushed back; candidates without a vector are never penalized:

using LexiSharp.Hybrid;

var vectors = new Dictionary<string, ReadOnlyMemory<float>>
{
    ["doc-1"] = embedding1, // pre-computed with your IEmbeddingProvider
    ["doc-2"] = embedding2,
};

IReranker mmr = new MaximalMarginalRelevanceReranker(vectors, lambda: 0.7, limit: 5);

Cascade chains any number of stages, trimming between stages so only the strongest candidates reach the expensive final ones; it is itself an IReranker, so cascades nest:

var pipeline = new CascadeRerankPipeline(
    new IReranker[] { lexicalReranker, mmr },
    new CascadeRerankOptions(StageLimit: 20, FinalLimit: 5, MinimumScore: 0.01));

Cross-encoder re-scores the shortlist with a pairwise model — a precision stage for cases where whole-corpus scoring would be too expensive (ColBERT-style late interaction, an LLM judge, ...). The model itself is a consumer-provided seam (ICrossEncoderScorer, same contract as IEmbeddingProvider: LexiSharp never runs the model):

IReranker cross = new CrossEncoderReranker(myOnnxCrossEncoder, limit: 5);

CrossEncoderReranker replaces each candidate's score with the model's, drops 0/NaN/infinity scores, and applies an optional MinimumScore and Limit.

MaxSim (ColBERT-style late interaction) re-scores the shortlist token-by-token instead of as a single embedding: every token of the query is embedded, each scores against the whole candidate's token embeddings (max similarity per query token, summed), so a query token never has to "average itself away" across the document:

using LexiSharp.Core;
using LexiSharp.Hybrid;

var tokenVectors = new Dictionary<string, IReadOnlyList<ReadOnlyMemory<float>>>
{
    ["doc-1"] = doc1TokenEmbeddings, // token embeddings pre-computed at index time
    ["doc-2"] = doc2TokenEmbeddings,
};

IReranker maxsim = new MaxSimReranker(
    myTokenEmbedder,                 // ITokenEmbeddingProvider (core seam, consumer-provided)
    tokenVectors,                    // doc-side token embeddings, pre-computed with the Passage role
    limit: 5,
    minimumScore: 0.0);

MaxSimReranker scores each candidate as Σₜ max_tok cosine(q_t, d_tok) — for each query token, the best cosine against any of the candidate's token embeddings (the query side is re-embedded with the Query role per search) — drops 0/NaN/infinity scores and ranks by total. It is the middle ground between whole-document cosine and the full pairwise pass of a cross-encoder.

Metadata filters

Gate the corpus with structured predicates over SearchDocument.Fields — every filter must hold (AND), and filtering happens before scoring:

var options = new SearchOptions(
    Limit: 10,
    Filters:
    [
        new MetadataFilter("kind", MetadataFilterOperator.Equal, "article"),
        new MetadataFilter("year", MetadataFilterOperator.GreaterThan, "2023"),
        new MetadataFilter("tags", MetadataFilterOperator.Contains, "nlp"),
    ]);

var results = engine.Search("vector search", options);

Comparisons are culture-invariant; greater/less-than go numeric when both sides parse as numbers, otherwise ordinal. Documents missing a field fail everything except NotEqual. Every backend honors the same contract: the in-memory engines evaluate the predicate before scoring, and the PostgreSQL backends (lexical, fuzzy, vector, sparse) push it down as a parameterized predicate over the fields jsonb column. ParadeDB's custom planner rejects those predicate shapes next to its BM25 operator, so it filters in C# over the whole match set instead (same semantics, no pushdown).

Phrase queries

Quote a segment of the query to require its terms at consecutive document positions:

var results = engine.Search("neural \"machine learning\"", new SearchOptions(Limit: 10));

Parsing happens before tokenization (QueryParser): double quotes are otherwise an ordinary separator for the tokenizer. In a mixed query the phrase is a hard corpus gate while the free terms around it only contribute to scoring — a document that matches the phrase comes back even if it lacks every free term, and a document that only has the free terms never does. Several quoted segments are AND-ed. A query made solely of empty quotes matches nothing. Phrase checks assume a plain token stream: n-gram tokenizers emit overlapping tokens and break the consecutive-position guarantee.

The SQL backends honor the same syntax natively: PostgreSQL through websearch_to_tsquery, ParadeDB through the ### phrase operator (on that backend only phrases shape the match set when quotes are present — free terms stay out of WHERE, exactly mirroring the stock engine's scoring-only role for them). An engine without phrase support (fuzzy, vector, sparse) rejects a quoted query with NotSupportedException rather than ignoring the quotes.

Highlighting

Mark where a query matched inside a document — same normalization as the index, so hits land on token boundaries even when the source text differs in case or accents:

using LexiSharp.Highlighting;

var terms = Tokenizer.Default.Tokenize(query);   // or QueryParser.Parse(query, tokenizer).AllTerms

string marked = TextHighlighter.HighlightFull(document.Text, terms, Tokenizer.Default);
// "The <em>quick</em> brown fox jumps over the lazy dog"

IReadOnlyList<HighlightSnippet> snippets = TextHighlighter.Highlight(
    document.Text, terms, Tokenizer.Default,
    new HighlightOptions { MaxSnippets = 2, Padding = 30 });

Matching goes through the tokenizer's span mode (ISpanTokenizer.TokenizeWithSpans, built into Tokenizer): each normalized term carries its [Start, Length) offsets into the source, so the query must be tokenized with the same tokenizer. Nearby matches cluster into one snippet, windows snap outward to word boundaries, and overlapping ranges (n-gram tokenizers) merge before tagging.

Prefix & fuzzy queries

Suffix a free-text atom to expand it against the index vocabulary at search time:

engine.Search("neural*");    // every indexed term starting with "neural"
engine.Search("catt~");      // within 1 edit: "cat", "cats", "catt", ...
engine.Search("catt~2");     // within 2 edits (count clamped to 0–2)

Expansion is a stock-engine feature: the atom's base is tokenized first, then matched against an IVocabularyIndex (InMemoryTextIndex implements it), keeping at most 64 terms per atom — highest document frequency first, then ordinal order. An index without vocabulary support falls back to the atom's literal base term, i.e. the behavior of a query without operators. Operators are only recognized when suffixed to word characters, never inside quoted phrases (a "machine*" phrase stays literal). They are a LexiSharp query syntax, so an engine that does not interpret them rejects such a query with NotSupportedException instead of silently treating the operator as plain text — IQuerySyntaxSupport.SupportedQueryFeatures exposes each engine's supported matrix (RankedTextSearchEngine: phrases + expansions; PostgreSQL lexical and ParadeDB: phrases; fuzzy/vector/sparse: plain queries only).

Synonyms

Register synonym edges once, then every free query term pulls in its direct synonyms:

using LexiSharp.Linguistics;

var synonyms = new SynonymMap()
    .Add("car", "auto")                       // one-way: "car" also searches "auto"
    .AddEquivalent("auto", "automobile");     // bidirectional group

ITextSearchEngine engine = new RankedTextSearchEngine(
    new InMemoryTextIndex(),
    new Bm25Scorer(),
    synonyms: synonyms);

Entries are tokenized with the engine's tokenizer at construction and each must reduce to exactly one term (otherwise the constructor throws). Expansion is one level deep and non-transitive — a synonym's own synonyms are never pulled in — and applies to free terms only: quoted phrases stay literal. Like the prefix/fuzzy operators, this is a stock-engine feature; the SQL backends do not currently rewrite queries.

Facets

Get value counts for UI refinements alongside the ranked page:

using LexiSharp.Core;

IFacetedSearchEngine engine = new RankedTextSearchEngine(index, new Bm25Scorer());

FacetedSearchResult page = engine.SearchWithFacets(
    "fast car",
    new SearchOptions(Limit: 10),
    facetFields: ["kind", "lang"]);

foreach (var bucket in page.Buckets)
    foreach (var value in bucket.Values)          // count desc, then value ordinal
        Console.WriteLine($"{bucket.Field}={value.Value}: {value.Count}");

Results is identical to Search for the same arguments. Counts cover every document that passes the metadata filters, the phrase gates and the score thresholds — the whole match set — independently of Offset/Limit, which only cut Results. A document missing a field does not count for it (faceting reads Fields only, not Category), and fields no matching document carries are omitted from Buckets. Currently stock-engine only.

Span-first API

The text entry points accept ReadOnlySpan<char>, so a query already living in a buffer need not be copied into a string first:

ReadOnlySpan<char> query = buffer.AsSpan(offset, length);

var parsed = QueryParser.Parse(query, Tokenizer.Default);
IReadOnlyList<SearchResult> hits = engine.Search(query, new SearchOptions(Limit: 10));
FacetedSearchResult page = engine.SearchWithFacets(query, facetFields: ["kind"]);
ScoreExplanation? why = engine.Explain("doc-1", query);

Every overload is equivalent to its string counterpart. Tokenizer runs the same pipeline directly over the span (SIMD ASCII runs, rune decoding only on the non-ASCII path); other implementers fall back to the default interface method, which copies the span and forwards. A null literal still binds to the string overload, so the span path is null-free.

Cost-based routing (RoutedSearchEngine)

When the same corpus is reachable through several engines — say a stock in-memory engine and a SQL backend — route each query to the one that will do the least work:

using LexiSharp.Core;

var router = new RoutedSearchEngine(new[]
{
    new RoutedEngine("memory", memoryEngine),   // implements IQueryCostProbe
    new RoutedEngine("postgres", pgEngine),     // no probe: last resort
});

IReadOnlyList<SearchResult> hits = router.Search("machine learning");

The router owns no index — it never writes, and its Index/Add/Remove/Clear throw NotSupportedException; populate the engines yourself. Each query runs on exactly one engine, so the scores are that engine's own (the router never mixes or renormalizes them across engines). The default CheapestByCandidateCountEstimator asks every engine implementing IQueryCostProbe for EstimateCandidateCount and picks the smallest — ties keep the earliest engine — while engines without the probe are only used when no costed engine exists. The stock engine's estimate is the sum of the literal query terms' document frequencies; pass a custom IQueryCostEstimator to route on engine priority, latency history or query shape instead.

The router is capability-preserving: SearchWithFacets, SearchWithDetails and Explain run on the engine the estimator selects for that query, and the router's own EstimateCandidateCount reports the smallest estimate across its engines (so a routed engine can itself be a candidate inside another router). Because the selection is per query, a capability the selected engine lacks throws NotSupportedException rather than silently re-routing. BoostedTextSearchEngine/RerankedTextSearchEngine also forward IQueryCostProbe to their inner engine (they do not change how many candidates a query touches); their score-mutating wrapper does not currently surface facets/detailed/explain, whose contracts it cannot preserve.

Intent-based routing (RoutingSearchEngine)

Cost routing picks the cheapest engine; intent routing picks the right one. Route each query to a pre-composed target — an engine plus optional metadata filters — chosen by a decision you supply (a rule, a small classifier, or a model behind IQueryRouter):

using LexiSharp.Core;

var router = new RoutingSearchEngine(
    new KeywordVsQuestionRouter(),                 // your IQueryRouter
    new[]
    {
        new SearchRoute("keywords", memoryEngine),
        new SearchRoute("questions", pgEngine, new[]
        {
            new MetadataFilter("kind", MetadataFilterOperator.Equal, "faq"),
        }),
    },
    fallbackId: "keywords",
    minimumConfidence: 0.6);

IReadOnlyList<SearchResult> hits = router.Search("how do I reset my password?");

The router only chooses among the ids it is given (RouteAsync(query, candidateIds)); it never builds filters or touches indexes. The selected route's filters are AND-ed onto the caller's SearchOptions.Filters. A null decision, an unknown id, a confidence below the threshold or a thrown exception all run the fallback route — a broken router never breaks a search. The seam is async (ValueTask<QueryRoute?>) because a model-backed router is naturally async, but the synchronous Search blocks on it, like the PostgreSQL engines block on IEmbeddingProvider. LexiSharp never runs a model itself: you provide the rule, classifier or model.

Lexical similarity and keyword extraction

Pairwise similarity for near-duplicate detection and record de-duplication — token-set measures over the library tokenizer, pg_trgm-style trigrams, and Levenshtein:

using LexiSharp.Similarity;

bool duplicate = LexicalSimilarity.Jaccard(stored, incoming) > 0.5;
double fuzzy   = LexicalSimilarity.Trigram("kubernetes cluster", "kubernetes clusters");
int edits      = LevenshteinDistance.Distance("kitten", "sitting"); // 3

Keyword extraction pulls the representative terms out of a text. TF-IDF becomes corpus-aware when built over an ITextIndex; TextRank needs no corpus at all:

using LexiSharp.Keywords;

IKeywordExtractor tags = new TfIdfKeywordExtractor(someIndex, StopWordTokenizer);
IKeywordExtractor graph = new TextRankKeywordExtractor(StopWordTokenizer); // co-occurrence + PageRank

foreach (var keyword in graph.Extract(document.Text, topN: 5))
    Console.WriteLine($"{keyword.Term}: {keyword.Score:F3}");

Index persistence (LexiSharp.MessagePack)

Save and reload an InMemoryTextIndex as compact, LZ4-compressed MessagePack binary — documents (id, text, fields, category) and tokenizer configuration:

// install once:  dotnet add package LexiSharp.MessagePack
using LexiSharp.MessagePack;

MessagePackTextIndexPersistence.Save(index, "corpus.bin");
var reloaded = MessagePackTextIndexPersistence.Load("corpus.bin"); // identical statistics, no re-indexing

A Tokenizer (stop words, n-grams, single-char terms) is reconstructed automatically. A custom ITokenizer is not currently serialized: hand the same implementation to Load — a type-name check protects against rebuilding with the wrong pipeline. Stemmed tokenizers likewise require the original tokenizer at load time (stemmers are not currently serializable).

The same package persists a sparse engine through MessagePackSparseIndexPersistence: the stored corpus is the documents plus their learned weights, so reloading bypasses the model — only queries need the ISparseEmbeddingProvider again:

MessagePackSparseIndexPersistence.Save(sparseEngine, "splade.bin");
var reloaded = MessagePackSparseIndexPersistence.Load("splade.bin", mySplade); // exact same search scores

Explainable scoring and BM25 tuning

Audit any ranking decision term by term, then let the corpus pick its own parameters. The explainers cover the additive scorers — BM25, TF-IDF and query likelihood (QueryLikelihoodScorer also exposes the collection-model contribution of query terms the document does not contain); a pure filter like BooleanScorer has no additive breakdown and Explain returns null for it:

var engine = new RankedTextSearchEngine(index, new TfIdfScorer());
ScoreExplanation? why = engine.Explain("doc-1", "search engine");
// why.Terms -> per-term TF, IDF and score contribution; why.LengthRatio, why.Parameters...

var bm25 = new RankedTextSearchEngine(index, new Bm25Scorer());
ScoreExplanation? whyBm25 = bm25.Explain("doc-1", "search engine");

var tuner = new Bm25ParameterTuner(index, validationQueries: [
    new Bm25ValidationQuery("search engine", ["doc-1", "doc-7"]),
    new Bm25ValidationQuery("fuzzy matching", ["doc-3"]),
]);
Bm25TuningResult tuning = tuner.Tune(topK: 5);          // grid search over k1 x b
var tunedEngine = new RankedTextSearchEngine(index, new Bm25Scorer(tuning.Parameters));

Each validation query lists the relevant document ids; candidates are judged with Precision@k, Recall@k, F1@k (default) or nDCG@k (RetrievalMetrics), averaged over the set. The index is never mutated; tuning.Grid exposes every evaluated (k1, b) point.

PostgreSQL backend (LexiSharp.Postgres)

Persistent, shared, concurrent search on top of a classic PostgreSQL setup. Several engines, all implementing ITextSearchEngine and sharing the same documents table (so the hybrid engine can fan out and merge lexical + vector + fuzzy results with ReciprocalRankFusionMerger):

Lexical (PostgresTextSearchEngine) — full-text over tsvector:

// install once:  dotnet add package LexiSharp.Postgres
using LexiSharp.Postgres;

ITextSearchEngine engine = new PostgresTextSearchEngine(
    "Host=db;Port=5432;Username=app;Password=secret;Database=search");

engine.Add(new SearchDocument("1", "the quick brown fox jumps over the lazy dog",
    new Dictionary<string, string> { ["kind"] = "fable" }, "fable"));

The provider installs (idempotently) the unaccent extension, a documents table with a tsv tsvector column and a GIN index, then queries it with websearch_to_tsquery and ranks with ts_rank_cd. Combined with the simple config, unaccent mirrors LexiSharp's accent-insensitive normalization. Scores are PostgreSQL-native, so they are not numerically comparable to Bm25Scorer/TfIdfScorer — feed both backends into the hybrid engine below when you need one consistent ordering.

Vector (PostgresVectorSearchEngine) — ANN over pgvector (HNSW or IVFFlat), fed by your own embeddings:

ITextSearchEngine vector = new PostgresVectorSearchEngine(
    "Host=db;Port=5432;Username=app;Password=secret;Database=search",
    new MyEmbeddingProvider(),                                // your ONNX/model-server deps, never LexiSharp
    new PostgresVectorOptions { Dimension = 384, Distance = VectorDistance.Cosine });

vector.Add(new SearchDocument("1", "the quick brown fox ..."));
vector.Search("a fast fox");   // scores: cosine→1-dist, L2→1/(1+dist), inner product→-dist

The engine installs (idempotently) the vector extension, adds an embedding vector(D) column and an HNSW (or IVFFlat) index on the same documents table. IVFFlat needs rows to cluster lists, so the index is created on the first EnsureSchema() call after your first inserts. ANN results are approximate: combine with HybridTextSearchEngine + RRF to trade recall for speed — exact cosine behavior is verified in the integration suite.

The embedding seam is role-aware: PostgresVectorSearchEngine embeds indexed documents with EmbeddingUse.Passage and queries with EmbeddingUse.Query, so asymmetric models (E5 prefixes and friends) work through the same single provider method. Finer knobs live in the options (see the class docs): HnswEfSearch (per-search candidate list; the engine wraps the query in a SET LOCAL hnsw.ef_search transaction), EmbeddingTextField (embed a named SearchDocument.TextFields entry instead of Text, leaving content/tsv untouched for the lexical engine) and, for a title/description (or any multi-section) split, EmbeddingColumns — one {suffix}_embedding vector(D) column and ANN index per configured TextFields entry. A search then targets a subset of columns (SearchWithColumns) or all of them, OR-fusing candidates by best per-column similarity, and SearchWithDetails (IDetailedSearchEngine) exposes each column's contribution for UI badges or telemetry. To encode the same query differently per column (e.g. center each channel's query in its own space), implement IColumnAwareEmbeddingProvider: the engine then passes the column label on every query and passage call; a plain IEmbeddingProvider keeps the single query vector shared across columns. The engine also implements IListableSearchEngine, so the full set of stored ids can be streamed (ListDocumentIds / ListDocumentIdsAsync, keyset pagination) to diff against an external ledger.

By default the integration tests are skipped unless POSTGRES_TEST_CONNECTION points at a live instance (e.g. Host=localhost;Port=5432;Username=postgres;Password=postgres;Database=lexisharp). The vector and sparse tests additionally require the vector extension: use the pgvector/pgvector:pg16 image (lexical tests only need stock PostgreSQL).

Sparse (PostgresSparseSearchEngine) — learned-sparse ANN over pgvector sparsevec (HNSW only — IVFFlat is unavailable for sparsevec), fed by your own sparse model:

ITextSearchEngine sparse = new PostgresSparseSearchEngine(
    connectionString,
    mySpladeProvider,                                       // ISparseEmbeddingProvider, never LexiSharp
    new PostgresSparseOptions
    {
        Vocabulary = vocabulary,                            // term → coordinate, fixed up front
        Distance = SparseDistance.InnerProduct,             // default; dot product suits SPLADE
    });

sparse.Add(new SearchDocument("1", "the quick brown fox ..."));
sparse.Search("a fast fox");   // scores: inner product→-dist, cosine→1-dist, L2/L1→1/(1+dist)

The engine installs (idempotently) the vector extension, adds a sparse sparsevec(D) column and an HNSW index on the same shared documents table. Because sparsevec is a positional format, the vocabulary (term → coordinate) is an index-layout decision: it must be fixed once and shared between the provider at index time and the one at query time. Terms outside the vocabulary are ignored; a query with no known term returns nothing. HNSW — unlike IVFFlat — works on empty tables and supports inserts, so there is no "index after first batch" step. sparsevec caps a vector at 1000 non-zero elements. Scores are PostgreSQL-native and again depend on the chosen distance; route through ReciprocalRankFusionMerger when mixing with the lexical engine.

Fuzzy (PostgresFuzzySearchEngine) — approximate, typo-tolerant matching over pg_trgm trigrams, with optional fuzzystrmatch (edit distance + phonetics):

ITextSearchEngine fuzzy = new PostgresFuzzySearchEngine(connectionString, new PostgresFuzzyOptions
{
    SearchMode = TrgmSearchMode.Nearest,        // kNN: closest labels first (autocomplete)
    // SearchMode = TrgmSearchMode.Similarity,  // threshold: content % query (de-dup, did-you-mean)
    SimilarityThreshold = 0.3,                  // honored via set_limit() in Similarity mode
    UseLevenshteinRefinement = true,            // exact edit-distance post-filter
    IncludePhonetic = true,                     // metaphone column; phonetic matches (Similarity mode)
});
fuzzy.Index(new[]
{
    new SearchDocument("1", "katherine"),
    new SearchDocument("2", "catherine"),
});
fuzzy.Search("caterin");   // typo-tolerant: both labels come back

The engine installs (idempotently) pg_trgm (+ fuzzystrmatch when enabled) and GiST and GIN trigram indexes on the same shared documents table. Scores are trigram similarities in [0, 1] (1 identical, 0 no shared trigram → excluded, honoring the library's score-0 convention). Nearest mode orders with the GiST kNN operator (content <-> query); Similarity mode ranks by similarity() above the configured threshold. These are the classic building blocks for autocomplete, de-duplication of names/addresses and "did you mean". PostgreSQL-native scores again call for ReciprocalRankFusionMerger when mixing with other engines.

ParadeDB BM25 backend (ParadeDBTextSearchEngine)

Okapi BM25 ranking computed by Tantivy inside PostgreSQL through the pg_search extension — an option to consider when ts_rank_cd ranking is not good enough and true BM25 is wanted. It ships in the same LexiSharp.Postgres package as the other PostgreSQL engines; only the pg_search extension (a self-hosted install or the ParadeDB image) is required server-side.

// install once:  dotnet add package LexiSharp.Postgres
using LexiSharp.ParadeDB;

ITextSearchEngine engine = new ParadeDBTextSearchEngine(connectionString);
engine.Add(new SearchDocument("1", "the quick brown fox jumps over the lazy dog"));

The engine installs (idempotently) the pg_search extension and a ParadeDB index (USING paradedb, the renamed USING bm25) on the same shared documents table as the Postgres engines, then matches with the ||| disjunction operator and ranks with pdb.score(id) — real BM25 (Tantivy variant), unlike ts_rank_cd. The default content tokenizer (pdb.simple with ASCII folding) mirrors LexiSharp's diacritic-insensitive, lowercase normalization:

new ParadeDBTextSearchEngine(
    connectionString,
    new ParadeDBOptions { ContentTokenizer = "pdb.simple('ascii_folding=true')" });

BM25 scores are PostgreSQL-native, so they are not numerically comparable to Bm25Scorer/TfIdfScorer — use the hybrid engine's ReciprocalRankFusionMerger (or re-rank) for a single cross-engine ordering. Note the extension is AGPL-3 licensed: fine for SaaS/internal use, but review it if you distribute the stack.

By default the tests are skipped unless POSTGRES_TEST_CONNECTION points at a live instance with pg_search available (the paradedb/paradedb:pg16 image ships it, preloaded) — they self-skip when the extension is absent.

Hybrid engine

Federate a hot in-memory index and a cold persistent backend, and produce one consistent global ranking:

using LexiSharp.Core;
using LexiSharp.Hybrid;
using LexiSharp.Indexing;
using LexiSharp.Ranking;

ITextSearchEngine hybrid = new HybridTextSearchEngine(new ITextSearchEngine[]
{
    new RankedTextSearchEngine(new InMemoryTextIndex(), new Bm25Scorer()), // hot subset
    postgresEngine,                                                       // cold backend
});

IReadOnlyList<SearchResult> results = hybrid.Search("textual search");

HybridTextSearchEngine queries every engine, de-duplicates the candidates by document id, then re-ranks the whole union with a single scorer (RerankingResultMerger, default BM25). Writes fan out to every engine. Three merge strategies are available:

Merger	Behavior	Best for
RerankingResultMerger (default)	re-scores the union with one ITextScorer	comparable stats, identical score scale wanted
ReciprocalRankFusionMerger	Σ 1/(k + rank) (k=60), rank-only	engines with incomparable scales — lexical + vector, ts_rank_cd vs BM25 (Postgres vs ParadeDB vs in-memory)
WeightedScoreResultMerger	normalized per-engine score blend	native scores trusted, per-engine weights wanted

Reciprocal Rank Fusion never looks at scores, so it bridges engines whose scores are not comparable — the sparse and dense embedding backends land in the same formula without calibration.

For score breakdowns, SearchWithDetails() returns DetailedSearchResults where each document also carries its raw per-source score (Contributions), keyed by the labels passed as sourceNames to the constructor ("lexical", "semantic", ... — default "engine-N"). A source that did not return the document is simply absent from that dictionary; the merged ordering from Search() is unchanged.

Embeddings are an agreed seam, not a feature here: IEmbeddingProvider (core) describes how a consumer project (ONNX model, model server, ...) would produce vectors — LexiSharp never computes embeddings — and VectorSimilarity provides pure cosine math. PostgresVectorSearchEngine is the reference consumer: it turns any provider into an ANN backend that the same HybridTextSearchEngine merges exactly like a lexical engine.

Sparse learned embeddings (ISparseEmbeddingProvider, SparseTextSearchEngine)

SPLADE-style models (SPLADE, uniCOIL, ...) produce sparse learned vectors: a handful of term → weight pairs where the weights are learned instead of tf-idf/BM25 frequencies. The key insight of this family is that it does not replace the inverted-index infrastructure, only the scoring function. SparseTextSearchEngine applies exactly that: it keeps a classic term → document → weight inverted index, and a query scores each document by sparse dot product Σ_t w_q(t)·w_d(t,d) — only the terms the learned model activated are visited:

using LexiSharp.Core;
using LexiSharp.Indexing;

ISparseEmbeddingProvider splade = myOnnxSplade; // consumer-provided, incl. vocab mapping
ITextSearchEngine engine = new SparseTextSearchEngine(splade);

engine.Add(new SearchDocument("doc-1", "sparse retrievers beat dense on exact terms"));

var results = engine.Search("learned sparse retrieval");

Like IEmbeddingProvider, ISparseEmbeddingProvider is a pure seam in the core: the ONNX model, tokenizer and vocabulary live in the consumer — and the EmbeddingUse role is threaded through uniformly across the dense/sparse/token seams (Passage at index time, Query per search), so asymmetric sparse variants keep a hook even though most SPLADE models are symmetric. A walkthrough of writing a SPLADE provider (ONNX + HuggingFace tokenizer + vocabulary mapping) is in docs/SPLADE.md — an outline, not a tested reference implementation. Weights are expected non-negative (ReLU-like); non-positive values are treated as "term absent". The engine implements ITextSearchEngine, so it drops straight into HybridTextSearchEngine where it merges with BM25 and dense engines via ReciprocalRankFusionMerger — RRF keeps sparse-only hits (matching terms the lexical scorer and the dense cosine disagree on) that a BM25 re-scoring merge would drop.

The engine never re-embeds on reload: Export() / Import() decouple inference from persistence, MessagePackSparseIndexPersistence serializes the stored weights directly, and PostgresSparseSearchEngine is the same model over pgvector sparsevec. In every backend, only queries keep needing the provider after the corpus is loaded.

Reranking stage

The reranking stage composes with all of it: wrap the hybrid in a RerankedTextSearchEngine (core decorator) and pass a CascadeRerankPipeline, a MaximalMarginalRelevanceReranker, a CrossEncoderReranker or a MaxSimReranker to add a precision or diversity pass on top of the fused ranking — the same two-stage retrieve-then-rerank shape, one line of composition.

Architecture

Package            Responsibilities
─────────────────────────────────────────────────────────────────────────────
LexiSharp         records + interfaces + in-memory index + scorers + tokenizer + IEmbeddingProvider + ISparseEmbeddingProvider + sparse engine (export/import) + boost/rerank decorators + filters + highlighting + facets + similarity + keywords + metrics + hybrid federation (RRF, weighted, cascade, cross-encoder, MMR, MaxSim)
LexiSharp.Postgres  PostgreSQL providers: tsvector+unaccent (lexical), pgvector ANN (vector), pgvector sparsevec (sparse), pg_trgm+fuzzystrmatch (fuzzy), pg_search/Tantivy (true BM25)
LexiSharp.MessagePack   MessagePack (binary) persistence for the in-memory index and the sparse engine

Within the core package, separation of concerns mirrors the recommendations the library was designed from:

• the index owns corpus statistics (tf, df, document length, positions, vocabulary);
• the scorer is a pure strategy reading from the index;
• the engine orchestrates query tokenization, scoring, filtering and ranking.

Scoring conventions

• A score of exactly 0 means not a match and the document is excluded from results (all built-in scorers honor this).
• Every sub-system is culture-agnostic; text is normalized to lowercase without accents so that "Résumé" and "resume" match.

Building & testing

dotnet build LexiSharp.slnx
dotnet test  tests/LexiSharp.Tests                # xUnit suite (Postgres tests need POSTGRES_TEST_CONNECTION)
dotnet run  --project bench/LexiSharp.Benchmarks  # BenchmarkDotNet suite (published numbers: BENCHMARKS.md)

Postgres/ParadeDB integration tests run against whatever POSTGRES_TEST_CONNECTION points to: pgvector/pgvector:pg16 covers the lexical + vector + sparse + fuzzy suites, paradedb/paradedb:pg16 covers the lexical + ParadeDB (BM25) + fuzzy suites. The sparse tests self-skip when the vector extension is unavailable. The fuzzy tests self-skip when pg_trgm (and fuzzystrmatch, when exercised) are unavailable.

License

MIT