web_search_helper.text module

Text normalisation shared by the index builder and the PHP engine.

The PHP endpoint tokenises the visitor’s question; this module tokenises the corpus at build time. The two must cut words the same way, or a word present in the corpus will never be found. Every rule here therefore has a twin in assets/search.php, and tests/test_php_parity.py runs both on the same strings.

>>> fold("Éléphant")
'elephant'
>>> tokenize("La parabole de l'éléphant", stopwords("fr"))
['parabole', 'elephant']
web_search_helper.text.STOPWORDS = {'en': 'the a an of and is are that which for in on by this these those his her its with without more less not or but so as when what how why there here it he she they we you be have do does did', 'fr': 'le la les un une des de du et est sont que qui pour dans sur par ce cet cette ces son sa ses avec sans plus moins pas ne ou mais donc car ni si comme quand tout tous toute toutes il elle ils elles nous vous etre avoir fait faire quel quelle quels quelles quoi dont cela ceci leur leurs pourquoi comment'}

BM25’s idf already discounts frequent words; these only remove words too frequent to mean anything in a question. Folded spellings are what count, since tokens are folded first.

Type:

Default stopword lists. Short on purpose

web_search_helper.text.fold(s)[source]

Lower-case, then strip accents (NFD, combining marks U+0300–U+036F).

>>> fold("Œuvre à Noël")
'œuvre a noel'
Parameters:

s (str)

Return type:

str

web_search_helper.text.stopwords(lang)[source]

Folded stopwords for lang; empty for a language without a list.

>>> "the" in stopwords("en"), len(stopwords("xx"))
(True, 0)
Parameters:

lang (str)

Return type:

frozenset[str]

web_search_helper.text.strip_math(s)[source]

Keep only the symbols of $$…$$ and \(…\) formulas.

>>> strip_math(r"a \(\mathbf{x}_i\) b")
'a x_i b'
Parameters:

s (str)

Return type:

str

web_search_helper.text.tokenize(s, stop=frozenset({}))[source]

Words of more than two characters, folded, stopwords removed.

>>> tokenize("What is the data wall?", stopwords("en"))
['data', 'wall']
Parameters:
Return type:

list[str]