web_search_helper.index module
The index file: everything the PHP engine needs, in one file it never serves.
The file is a PHP script that, requested by URL, answers 404 and exits. The
data follows __halt_compiler();, which PHP never parses. The corpus is
therefore served in clear at no address, even when the web server ignores the
.htaccess that also denies the folder (an FTP client that skips dotfiles
is enough for that).
Layout after the header (little-endian):
|
magic |
u32 + JSON |
metadata: config, thresholds, passages, section of each passage |
u32 × 4 |
number of vectors, sections, words, bytes of postings |
f32 |
window vectors, |
f32 |
section centroids, |
u32 + str |
the lexicon, words joined by |
u32 |
|
(u32,u16) |
postings: (passage, term frequency), passages in increasing order |
u32 |
length of each passage in tokens |
Everything that can be computed once is computed here, so that a request only reads: the lexical index, the section centroids, the window offsets.
>>> from web_search_helper.corpus import Passage
>>> from web_search_helper.embed import HashEmbedder
>>> idx = build([Passage("Un éléphant dans une pièce.", "Doc", "S")], HashEmbedder(), "fr")
>>> data = to_bytes(idx)
>>> data.startswith(HEADER), from_bytes(data).passages[0]["t"]
(True, 'Un éléphant dans une pièce.')
- web_search_helper.index.DEFAULT_THRESHOLDS = {'df_min': 9, 'lex_min': 5.0, 'sec_min': 0.561, 'sem_min': 0.55}
Thresholds of the “I cannot answer” rule. Defaults are those measured on the corpus this engine was first built for; calibrate measures yours.
- web_search_helper.index.HEADER = b'<?php http_response_code(404); exit; __halt_compiler();'
Requested by URL, the file answers 404 and stops.
search.phpchecks the same bytes before reading.
- class web_search_helper.index.Index(meta: 'dict', passages: 'list[dict]', section_of: 'list[int]', vectors: 'np.ndarray', centroids: 'np.ndarray', words: 'list[str]', postings: 'dict[str, list[tuple[int, int]]]', lengths: 'list[int]', first_vec: 'list[int]' = <factory>)[source]
Bases:
object- Parameters:
- centroids: ndarray
- vectors: ndarray
- web_search_helper.index.build(passages, embedder, lang, stop=None, thresholds=None, limits=None, ranking=None, batch=256)[source]
Cut, embed and index
passages.
- web_search_helper.index.from_vectors(rows, vectors, lang, model, stop=None, thresholds=None, limits=None, ranking=None)[source]
Index passages whose windows are already embedded, by another pipeline.
For a site that computes its own embeddings and wants the engine without re-embedding (or wants the SAME vectors, to compare engines).
rowsare dicts witht(text),c(document),s(section),o(end offset of each window int) and optionallyu(url) andm(meta passed to the browser);vectorsholds one L2-normalised row per window, passages in order.modelnames the Python model and its transformers.js twin, which the page must use for the question.>>> rows = [{"t": "Un éléphant.", "c": "D", "s": "S", "o": [12]}] >>> idx = from_vectors(rows, np.eye(1, 4, dtype=np.float32), "fr", ... {"python": "m", "browser": "Xenova/m"}) >>> idx.meta["dim"], idx.words (4, ['elephant'])