web_search_helper.index module

The index file: everything the PHP engine needs, in one file it never serves.

The file is a PHP script that, requested by URL, answers 404 and exits. The data follows __halt_compiler();, which PHP never parses. The corpus is therefore served in clear at no address, even when the web server ignores the .htaccess that also denies the folder (an FTP client that skips dotfiles is enough for that).

Layout after the header (little-endian):

WSH1

magic

u32 + JSON

metadata: config, thresholds, passages, section of each passage

u32 × 4

number of vectors, sections, words, bytes of postings

f32

window vectors, n_vec × dim, passages in order

f32

section centroids, n_sec × dim

u32 + str

the lexicon, words joined by \n, in order of first appearance

u32

n_words + 1 offsets into the postings

(u32,u16)

postings: (passage, term frequency), passages in increasing order

u32

length of each passage in tokens

Everything that can be computed once is computed here, so that a request only reads: the lexical index, the section centroids, the window offsets.

>>> from web_search_helper.corpus import Passage
>>> from web_search_helper.embed import HashEmbedder
>>> idx = build([Passage("Un éléphant dans une pièce.", "Doc", "S")], HashEmbedder(), "fr")
>>> data = to_bytes(idx)
>>> data.startswith(HEADER), from_bytes(data).passages[0]["t"]
(True, 'Un éléphant dans une pièce.')
web_search_helper.index.DEFAULT_THRESHOLDS = {'df_min': 9, 'lex_min': 5.0, 'sec_min': 0.561, 'sem_min': 0.55}

Thresholds of the “I cannot answer” rule. Defaults are those measured on the corpus this engine was first built for; calibrate measures yours.

web_search_helper.index.HEADER = b'<?php http_response_code(404); exit; __halt_compiler();'

Requested by URL, the file answers 404 and stops. search.php checks the same bytes before reading.

class web_search_helper.index.Index(meta: 'dict', passages: 'list[dict]', section_of: 'list[int]', vectors: 'np.ndarray', centroids: 'np.ndarray', words: 'list[str]', postings: 'dict[str, list[tuple[int, int]]]', lengths: 'list[int]', first_vec: 'list[int]' = <factory>)[source]

Bases: object

Parameters:
centroids: ndarray
first_vec: list[int]
property lang: str
lengths: list[int]
meta: dict
passages: list[dict]
postings: dict[str, list[tuple[int, int]]]
section_of: list[int]
property stop: frozenset[str]
vectors: ndarray
words: list[str]
web_search_helper.index.build(passages, embedder, lang, stop=None, thresholds=None, limits=None, ranking=None, batch=256)[source]

Cut, embed and index passages.

Parameters:
Return type:

Index

web_search_helper.index.from_bytes(data)[source]
Parameters:

data (bytes)

Return type:

Index

web_search_helper.index.from_vectors(rows, vectors, lang, model, stop=None, thresholds=None, limits=None, ranking=None)[source]

Index passages whose windows are already embedded, by another pipeline.

For a site that computes its own embeddings and wants the engine without re-embedding (or wants the SAME vectors, to compare engines). rows are dicts with t (text), c (document), s (section), o (end offset of each window in t) and optionally u (url) and m (meta passed to the browser); vectors holds one L2-normalised row per window, passages in order. model names the Python model and its transformers.js twin, which the page must use for the question.

>>> rows = [{"t": "Un éléphant.", "c": "D", "s": "S", "o": [12]}]
>>> idx = from_vectors(rows, np.eye(1, 4, dtype=np.float32), "fr",
...                    {"python": "m", "browser": "Xenova/m"})
>>> idx.meta["dim"], idx.words
(4, ['elephant'])
Parameters:
Return type:

Index

web_search_helper.index.load(path)[source]
Parameters:

path (str | Path)

Return type:

Index

web_search_helper.index.save(idx, path)[source]
Parameters:
Return type:

Path

web_search_helper.index.to_bytes(idx)[source]
Parameters:

idx (Index)

Return type:

bytes

web_search_helper.index.vectors_from_int8(data, dim)[source]

int8 vectors (component × 127), read back as a browser would: each row divided by its own norm, in float32.

>>> vectors_from_int8(bytes([127, 0, 0, 0]), 4).tolist()
[[1.0, 0.0, 0.0, 0.0]]
Parameters:
Return type:

ndarray