web_search_helper.engine module

The search, in Python: the reference twin of assets/search.php.

The PHP file is what answers visitors; this module answers the same question the same way, so that thresholds can be measured (calibrate) and the PHP engine checked against it (tests/test_php_parity.py). Any change to the ranking goes into BOTH, or the parity test fails.

Three signals, two engines, one verdict:

  • words: BM25 (k1 = 1.5, b = 0.75) with a conjunctive pass first (if some passages contain every word of the question, only they compete), a coverage² weight otherwise, and a timid typo tolerance (a word is replaced by its neighbours only when it is absent from the corpus);

  • meaning: the cosine of each passage’s best window, and the cosine of the best section centroid;

  • fusion: reciprocal rank fusion of the two rankings (k = 60);

  • verdict: answer only if one of three routes says the corpus treats the question; otherwise return nothing at all.

web_search_helper.engine.MATH = re.compile('\\$\\$.*?\\$\\$|\\\\\\[.*?\\\\\\]|\\\\\\(.*?\\\\\\)', re.DOTALL)

display $$…$$ and […], inline (…).

Type:

TeX formulas an excerpt must never cut

web_search_helper.engine.excerpt(i, qv, terms, idx)[source]

Two windows of passage i, snapped to sentence boundaries.

The best window and its better neighbour: a few sentences, never the whole passage. The start moves forward to the first whole sentence; the end moves forward to the end of the current sentence (at most 400 characters). Window bounds and cosines travel with the excerpt: the page needs them to compose its extractive answer.

Parameters:
Return type:

dict

web_search_helper.engine.expand_terms(q_terms, idx)[source]

One group per question word: the word itself, or up to three neighbours.

Neighbours rank by corpus frequency over (1 + distance)², and keep only those weighing at least a quarter of the best. A word unknown even from afar keeps an EMPTY group: it still counts against coverage and blocks the conjunctive pass, so an off-topic question does not shrink to the one word the corpus happens to use.

Parameters:
Return type:

list[list[tuple[str, float]]]

web_search_helper.engine.lexical(q, idx)[source]
Parameters:
Return type:

dict

web_search_helper.engine.max_typos(w)[source]
Parameters:

w (str)

Return type:

int

web_search_helper.engine.rrf(a, b, k, top)[source]
Parameters:
Return type:

list[int]

web_search_helper.engine.search(idx, q, qv=None, explain=False)[source]

What search.php answers to q (and the question vector qv).

Parameters:
Return type:

dict

web_search_helper.engine.semantic(qv, idx)[source]
Parameters:
  • qv (ndarray)

  • idx (Index)

Return type:

dict

web_search_helper.engine.sure_enough(kw, sem, th)[source]

The verdict: three routes in OR, because none suffices everywhere.

  • one typed word: only its PRESENCE decides (it must appear in at least df_min passages); its embedding is too poor to judge;

  • meaning: best window ≥ sem_min AND best section ≥ sec_min;

  • words: every word found in one passage AND its BM25 ≥ lex_min.

Parameters:
Return type:

bool

web_search_helper.engine.within(a, b, budget)[source]

Levenshtein distance, or budget + 1 as soon as it is exceeded.

>>> within("elefant", "elephant", 2), within("art", "zzzzzz", 1)
(2, 2)
Parameters:
Return type:

int