web_search_helper.engine module
The search, in Python: the reference twin of assets/search.php.
The PHP file is what answers visitors; this module answers the same question
the same way, so that thresholds can be measured (calibrate) and the PHP
engine checked against it (tests/test_php_parity.py). Any change to the
ranking goes into BOTH, or the parity test fails.
Three signals, two engines, one verdict:
words: BM25 (k1 = 1.5, b = 0.75) with a conjunctive pass first (if some passages contain every word of the question, only they compete), a coverage² weight otherwise, and a timid typo tolerance (a word is replaced by its neighbours only when it is absent from the corpus);
meaning: the cosine of each passage’s best window, and the cosine of the best section centroid;
fusion: reciprocal rank fusion of the two rankings (k = 60);
verdict: answer only if one of three routes says the corpus treats the question; otherwise return nothing at all.
- web_search_helper.engine.MATH = re.compile('\\$\\$.*?\\$\\$|\\\\\\[.*?\\\\\\]|\\\\\\(.*?\\\\\\)', re.DOTALL)
display $$…$$ and […], inline (…).
- Type:
TeX formulas an excerpt must never cut
- web_search_helper.engine.excerpt(i, qv, terms, idx)[source]
Two windows of passage
i, snapped to sentence boundaries.The best window and its better neighbour: a few sentences, never the whole passage. The start moves forward to the first whole sentence; the end moves forward to the end of the current sentence (at most 400 characters). Window bounds and cosines travel with the excerpt: the page needs them to compose its extractive answer.
- web_search_helper.engine.expand_terms(q_terms, idx)[source]
One group per question word: the word itself, or up to three neighbours.
Neighbours rank by corpus frequency over (1 + distance)², and keep only those weighing at least a quarter of the best. A word unknown even from afar keeps an EMPTY group: it still counts against coverage and blocks the conjunctive pass, so an off-topic question does not shrink to the one word the corpus happens to use.
- web_search_helper.engine.search(idx, q, qv=None, explain=False)[source]
What
search.phpanswers toq(and the question vectorqv).
- web_search_helper.engine.sure_enough(kw, sem, th)[source]
The verdict: three routes in OR, because none suffices everywhere.
one typed word: only its PRESENCE decides (it must appear in at least
df_minpassages); its embedding is too poor to judge;meaning: best window ≥
sem_minAND best section ≥sec_min;words: every word found in one passage AND its BM25 ≥
lex_min.