web_search_helper package

Submodules

Module contents

web-search-helper — search a private corpus from a website, PHP only.

Build the index on your machine; serve it with one PHP file; embed the question in the browser. The corpus is never published: the page receives short excerpts of what answers the question, or nothing when the corpus does not treat it.

>>> import web_search_helper as wsh
>>> callable(wsh.build) and callable(wsh.search)
True
class web_search_helper.HashEmbedder(dim=64, window_words=40)[source]

Bases: object

Deterministic bag-of-words vectors, for tests and offline dry runs.

Not a semantic model: two texts are close when they share words. It lets the whole pipeline, PHP included, be tested without downloading weights.

Parameters:
  • dim (int)

  • window_words (int)

browser = 'hash'
encode(texts)[source]
Parameters:

texts (list[str])

Return type:

ndarray

property name: str
windows(text)[source]
Parameters:

text (str)

Return type:

list[int]

class web_search_helper.Index(meta: 'dict', passages: 'list[dict]', section_of: 'list[int]', vectors: 'np.ndarray', centroids: 'np.ndarray', words: 'list[str]', postings: 'dict[str, list[tuple[int, int]]]', lengths: 'list[int]', first_vec: 'list[int]' = <factory>)[source]

Bases: object

Parameters:
centroids: ndarray
first_vec: list[int]
property lang: str
lengths: list[int]
meta: dict
passages: list[dict]
postings: dict[str, list[tuple[int, int]]]
section_of: list[int]
property stop: frozenset[str]
vectors: ndarray
words: list[str]
class web_search_helper.Passage(text: 'str', doc: 'str' = '', section: 'str' = '', url: 'str' = '', meta: 'dict' = <factory>)[source]

Bases: object

Parameters:
doc: str = ''
meta: dict
section: str = ''
text: str
to_dict()[source]
Return type:

dict

url: str = ''
class web_search_helper.SentenceTransformerEmbedder(name='paraphrase-multilingual-MiniLM-L12-v2', browser=None)[source]

Bases: object

A sentence-transformers model, cut into windows by its own tokenizer.

Parameters:
  • name (str)

  • browser (str | None)

encode(texts)[source]
Parameters:

texts (list[str])

Return type:

ndarray

windows(text)[source]
Parameters:

text (str)

Return type:

list[int]

web_search_helper.bench(index, runs=5, question='test question', php='php')[source]

Time the PHP engine on an index: load, then full searches (words + meaning).

The search is exhaustive (every window is compared with the question). ann-router would pick HNSW from 1 000 vectors under a 10 ms budget, but its thresholds are measured on numpy, not PHP, and the “I cannot answer” rule judges the ABSOLUTE best cosine, which an approximate index may miss. So the exhaustive scan stays as long as this measure fits the budget.

Parameters:
Return type:

dict

web_search_helper.build(passages, embedder, lang, stop=None, thresholds=None, limits=None, ranking=None, batch=256)[source]

Cut, embed and index passages.

Parameters:
Return type:

Index

web_search_helper.calibrate(idx, inside, outside, embedder)[source]

New thresholds and a report; idx.meta is left untouched.

Parameters:
Return type:

dict

web_search_helper.check_online(base_url, langs, question='test', data_dir='search-data')[source]

Is the corpus private, and does the endpoint answer?

Parameters:
Return type:

dict

web_search_helper.from_vectors(rows, vectors, lang, model, stop=None, thresholds=None, limits=None, ranking=None)[source]

Index passages whose windows are already embedded, by another pipeline.

For a site that computes its own embeddings and wants the engine without re-embedding (or wants the SAME vectors, to compare engines). rows are dicts with t (text), c (document), s (section), o (end offset of each window in t) and optionally u (url) and m (meta passed to the browser); vectors holds one L2-normalised row per window, passages in order. model names the Python model and its transformers.js twin, which the page must use for the question.

>>> rows = [{"t": "Un éléphant.", "c": "D", "s": "S", "o": [12]}]
>>> idx = from_vectors(rows, np.eye(1, 4, dtype=np.float32), "fr",
...                    {"python": "m", "browser": "Xenova/m"})
>>> idx.meta["dim"], idx.words
(4, ['elephant'])
Parameters:
Return type:

Index

web_search_helper.install(site, indexes, data_dir='search-data', demo=False)[source]

Copy the endpoint, the client and the index files into site.

demo also copies the bilingual demo page as demo.html.

Parameters:
Return type:

list[Path]

web_search_helper.load(path)[source]
Parameters:

path (str | Path)

Return type:

Index

web_search_helper.read_corpus(path, target_words=220)[source]

A .jsonl file, a single Markdown/text file, or a folder of them.

Parameters:
Return type:

list[Passage]

web_search_helper.read_jsonl(path)[source]
Parameters:

path (str | Path)

Return type:

list[Passage]

web_search_helper.save(idx, path)[source]
Parameters:
Return type:

Path

web_search_helper.search(idx, q, qv=None, explain=False)[source]

What search.php answers to q (and the question vector qv).

Parameters:
Return type:

dict

web_search_helper.serve(site, port=8765, php='php')[source]

php -S on site: the local twin of the hosting.

Parameters:
Return type:

Popen

web_search_helper.vectors_from_int8(data, dim)[source]

int8 vectors (component × 127), read back as a browser would: each row divided by its own norm, in float32.

>>> vectors_from_int8(bytes([127, 0, 0, 0]), 4).tolist()
[[1.0, 0.0, 0.0, 0.0]]
Parameters:
Return type:

ndarray