web_search_helper.embed module
Embedders: one model on the build machine, its twin in the browser.
The index is embedded here, with sentence-transformers; the visitor’s question
is embedded in the page, with transformers.js. Both must be the same model,
or cosines compare vectors from two different spaces. MODELS pairs each
supported Python name with its browser (ONNX) twin.
Long passages are cut into WINDOWS that the model reads entirely (it ignores everything past its token limit), and a passage scores the best cosine of its windows. The end offset of each window is kept: it tells the page WHERE in a passage the answer is, and it bounds the excerpt the server sends back.
- class web_search_helper.embed.Embedder(*args, **kwargs)[source]
Bases:
Protocol
- class web_search_helper.embed.HashEmbedder(dim=64, window_words=40)[source]
Bases:
objectDeterministic bag-of-words vectors, for tests and offline dry runs.
Not a semantic model: two texts are close when they share words. It lets the whole pipeline, PHP included, be tested without downloading weights.
- browser = 'hash'
- web_search_helper.embed.MODELS = {'all-MiniLM-L6-v2': 'Xenova/all-MiniLM-L6-v2', 'paraphrase-multilingual-MiniLM-L12-v2': 'Xenova/paraphrase-multilingual-MiniLM-L12-v2', 'sentence-transformers/all-MiniLM-L6-v2': 'Xenova/all-MiniLM-L6-v2', 'sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2': 'Xenova/paraphrase-multilingual-MiniLM-L12-v2'}
Python model -> transformers.js model of the same weights.
- class web_search_helper.embed.SentenceTransformerEmbedder(name='paraphrase-multilingual-MiniLM-L12-v2', browser=None)[source]
Bases:
objectA sentence-transformers model, cut into windows by its own tokenizer.