mod index

module index

Tantivy-backed full-text index for search and near-duplicate detection.

Functions

fn default_index_dir() -> PathBuf

Default index directory (gitignored) under the working tree.

fn find_duplicates(index_dir: &Path, corpus_dir: &Path, score_threshold: f64, per_doc_limit: usize) -> Result<Vec<DedupePair>>

Find near-duplicate pairs using Tantivy self-queries plus optional Jaccard floor.

For each document, query the index with a content-derived query and flag other documents that rank above score_threshold (excluding self). Pairs are unique and ordered by score desc.

fn find_duplicates_jaccard(docs: &[CannedResponse], threshold: f64) -> Vec<DedupePair>

Pure near-dup detection on an in-memory list via Jaccard (no Tantivy I/O).

fn jaccard_similarity(a: &str, b: &str) -> f64

Jaccard similarity over token sets (pure; used as a second signal for dedupe tests).

fn reindex(corpus_dir: &Path, index_dir: &Path) -> Result<usize>

Reindex all markdown responses under corpus_dir into Tantivy at index_dir. Returns number of documents written.

fn sanitize_query(query: &str) -> String

Strip characters that break Tantivy’s query parser; keep alphanumeric terms.

fn search(index_dir: &Path, query: &str, limit: usize) -> Result<Vec<SearchHit>>

Search the Tantivy index with BM25. Returns hits sorted by score desc.

fn self_query_for(doc: &CannedResponse) -> String

Build a short self-query for near-duplicate search from a response.

fn tokenize(text: &str) -> Vec<String>

Tokenize for pure lexical helpers (tests / jaccard).

Structs and Unions

struct DedupePair

A pair of near-duplicate responses for curation.

left_id: String

right_id: String

score: f64

left_path: String

right_path: String

reason: String

struct IndexDoc

One document stored in the search index.

id: String

title: String

content: String

path: String

tags: Vec<String>

sop: Option<String>

Traits implemented

impl From<&CannedResponse> for IndexDoc

struct SearchHit

A ranked hit from Tantivy BM25 search.

id: String

score: f64

title: String

snippet: String

path: PathBuf