web_search_helper.text module
Text normalisation shared by the index builder and the PHP engine.
The PHP endpoint tokenises the visitor’s question; this module tokenises the
corpus at build time. The two must cut words the same way, or a word present
in the corpus will never be found. Every rule here therefore has a twin in
assets/search.php, and tests/test_php_parity.py runs both on the same
strings.
>>> fold("Éléphant")
'elephant'
>>> tokenize("La parabole de l'éléphant", stopwords("fr"))
['parabole', 'elephant']
- web_search_helper.text.STOPWORDS = {'en': 'the a an of and is are that which for in on by this these those his her its with without more less not or but so as when what how why there here it he she they we you be have do does did', 'fr': 'le la les un une des de du et est sont que qui pour dans sur par ce cet cette ces son sa ses avec sans plus moins pas ne ou mais donc car ni si comme quand tout tous toute toutes il elle ils elles nous vous etre avoir fait faire quel quelle quels quelles quoi dont cela ceci leur leurs pourquoi comment'}
BM25’s idf already discounts frequent words; these only remove words too frequent to mean anything in a question. Folded spellings are what count, since tokens are folded first.
- Type:
Default stopword lists. Short on purpose
- web_search_helper.text.fold(s)[source]
Lower-case, then strip accents (NFD, combining marks U+0300–U+036F).
>>> fold("Œuvre à Noël") 'œuvre a noel'
- web_search_helper.text.stopwords(lang)[source]
Folded stopwords for
lang; empty for a language without a list.>>> "the" in stopwords("en"), len(stopwords("xx")) (True, 0)