web_search_helper.embed module

Embedders: one model on the build machine, its twin in the browser.

The index is embedded here, with sentence-transformers; the visitor’s question is embedded in the page, with transformers.js. Both must be the same model, or cosines compare vectors from two different spaces. MODELS pairs each supported Python name with its browser (ONNX) twin.

Long passages are cut into WINDOWS that the model reads entirely (it ignores everything past its token limit), and a passage scores the best cosine of its windows. The end offset of each window is kept: it tells the page WHERE in a passage the answer is, and it bounds the excerpt the server sends back.

class web_search_helper.embed.Embedder(*args, **kwargs)[source]

Bases: Protocol

browser: str
dim: int
encode(texts)[source]

L2-normalised float32 vectors, one row per text.

Parameters:

texts (list[str])

Return type:

ndarray

name: str
windows(text)[source]

End offsets (in characters) of the windows covering text.

Parameters:

text (str)

Return type:

list[int]

class web_search_helper.embed.HashEmbedder(dim=64, window_words=40)[source]

Bases: object

Deterministic bag-of-words vectors, for tests and offline dry runs.

Not a semantic model: two texts are close when they share words. It lets the whole pipeline, PHP included, be tested without downloading weights.

Parameters:
  • dim (int)

  • window_words (int)

browser = 'hash'
encode(texts)[source]
Parameters:

texts (list[str])

Return type:

ndarray

property name: str
windows(text)[source]
Parameters:

text (str)

Return type:

list[int]

web_search_helper.embed.MODELS = {'all-MiniLM-L6-v2': 'Xenova/all-MiniLM-L6-v2', 'paraphrase-multilingual-MiniLM-L12-v2': 'Xenova/paraphrase-multilingual-MiniLM-L12-v2', 'sentence-transformers/all-MiniLM-L6-v2': 'Xenova/all-MiniLM-L6-v2', 'sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2': 'Xenova/paraphrase-multilingual-MiniLM-L12-v2'}

Python model -> transformers.js model of the same weights.

class web_search_helper.embed.SentenceTransformerEmbedder(name='paraphrase-multilingual-MiniLM-L12-v2', browser=None)[source]

Bases: object

A sentence-transformers model, cut into windows by its own tokenizer.

Parameters:
  • name (str)

  • browser (str | None)

encode(texts)[source]
Parameters:

texts (list[str])

Return type:

ndarray

windows(text)[source]
Parameters:

text (str)

Return type:

list[int]

web_search_helper.embed.make_embedder(name, browser=None)[source]

The embedder an index records as meta["model"], rebuilt.

>>> make_embedder("hash:32:12").dim
32
Parameters:
  • name (str)

  • browser (str | None)

web_search_helper.embed.window_texts(text, ends)[source]

The text of each window, formulas reduced to their symbols.

Parameters:
Return type:

list[str]