Expand description
Tantivy-backed full-text index for search and near-duplicate detection.
Structs§
- Dedupe
Pair - A pair of near-duplicate responses for curation.
- Fields 🔒
- Index
Doc - One document stored in the search index.
- Search
Hit - A ranked hit from Tantivy BM25 search.
Functions§
- build_
schema 🔒 - default_
index_ dir - Default index directory (gitignored) under the working tree.
- field_
text 🔒 - find_
duplicates - Find near-duplicate pairs using Tantivy self-queries plus optional Jaccard floor.
- find_
duplicates_ jaccard - Pure near-dup detection on an in-memory list via Jaccard (no Tantivy I/O).
- jaccard_
similarity - Jaccard similarity over token sets (pure; used as a second signal for dedupe tests).
- open_
index 🔒 - reindex
- Reindex all markdown responses under
corpus_dirinto Tantivy atindex_dir. Returns number of documents written. - sanitize_
query - Strip characters that break Tantivy’s query parser; keep alphanumeric terms.
- search
- Search the Tantivy index with BM25. Returns hits sorted by score desc.
- self_
query_ for - Build a short self-query for near-duplicate search from a response.
- snippet_
for 🔒 - tokenize
- Tokenize for pure lexical helpers (tests / jaccard).