Semantic Modum algorithms

• Scope: inspected commit e63cced.

1. Ingestion — source

1. Read .json files directly inside MODUM_SCRAPE_DIR, defaulting to ../modum_scrape.
2. Load each document’s title, URL, path, already-generated embedding, and referenced text file.
3. Use the JSON filename as the document ID.
4. Derive category from the first path segment.
5. Derive date from YYYY-MM-DD in the path; otherwise use last_updated.
6. Calculate and cache each embedding’s vector length.
7. Skip malformed JSON; use empty text when the text file is unavailable.
8. Cache the dataset in server memory, without automatic refresh.

• Not included: scraping, content cleaning, chunking, document-embedding generation, or SiteVision synchronization. Those upstream algorithms aren’t in this repository.

2. Keyword scoring — source

1. Combine each document’s title and body.
2. Lowercase, remove combining accents—including converting å to a—split into words, then remove hardcoded stopwords.
3. Rebuild word frequencies, document lengths, and document frequencies for each search.
4. Calculate BM25, using k1 = 1.2, b = 0.75, and IDF = ln(1 + (N − df + 0.5)/(df + 0.5)).
5. Add 1.0 if the normalized query appears verbatim in title/body, provided phrase boosting is enabled and the query has ≥4 characters.
6. If fuzzy matching is enabled: add 0.15 per query token having a document word with character-trigram Jaccard similarity ≥0.6.
7. Add 0.6 per query token found in the title.
8. Add 0.5 × the longest consecutive matching token sequence between query and title.
9. Keep positive scores and sort descending.

3. Semantic scoring — source

1. Embed the trimmed query using the configured Azure OpenAI embedding deployment.
2. Compare it against every document embedding using cosine similarity.
3. Exclude documents with missing/incompatible embeddings, cosine ≤0, or cosine below the configured minimum.
4. Start with score = cosine.
5. Add the keyword contribution: min(0.3, keywordScore × 0.1).
6. If enabled, add recency: 0.2 × exp(−max(0, ageDays)/180); missing dates add zero.
7. If enabled, add 0.15 for a selected category.
8. Sort descending; no additional model reranking.

4. Current defaults and output — source

• Keyword blending and phrase boosting: on.
• Fuzzy matching, recency, and category boosts: off.
• Default final score: cosine + min(0.3, keywordScore × 0.1).
• Return separate keyword and semantic result lists: 20 results each by default, maximum 50.
• Snippet: first 240 characters of document text, rather than the matching passage.