web_search_helper package
Submodules
- web_search_helper.api module
- web_search_helper.calibrate module
- web_search_helper.cli module
- web_search_helper.corpus module
- web_search_helper.embed module
- web_search_helper.engine module
- web_search_helper.index module
- web_search_helper.mcp_server module
- web_search_helper.site module
- web_search_helper.text module
Module contents
web-search-helper — search a private corpus from a website, PHP only.
Build the index on your machine; serve it with one PHP file; embed the question in the browser. The corpus is never published: the page receives short excerpts of what answers the question, or nothing when the corpus does not treat it.
>>> import web_search_helper as wsh
>>> callable(wsh.build) and callable(wsh.search)
True
- class web_search_helper.HashEmbedder(dim=64, window_words=40)[source]
Bases:
objectDeterministic bag-of-words vectors, for tests and offline dry runs.
Not a semantic model: two texts are close when they share words. It lets the whole pipeline, PHP included, be tested without downloading weights.
- browser = 'hash'
- class web_search_helper.Index(meta: 'dict', passages: 'list[dict]', section_of: 'list[int]', vectors: 'np.ndarray', centroids: 'np.ndarray', words: 'list[str]', postings: 'dict[str, list[tuple[int, int]]]', lengths: 'list[int]', first_vec: 'list[int]' = <factory>)[source]
Bases:
object- Parameters:
- centroids: ndarray
- vectors: ndarray
- class web_search_helper.Passage(text: 'str', doc: 'str' = '', section: 'str' = '', url: 'str' = '', meta: 'dict' = <factory>)[source]
Bases:
object
- class web_search_helper.SentenceTransformerEmbedder(name='paraphrase-multilingual-MiniLM-L12-v2', browser=None)[source]
Bases:
objectA sentence-transformers model, cut into windows by its own tokenizer.
- web_search_helper.bench(index, runs=5, question='test question', php='php')[source]
Time the PHP engine on an index: load, then full searches (words + meaning).
The search is exhaustive (every window is compared with the question).
ann-routerwould pick HNSW from 1 000 vectors under a 10 ms budget, but its thresholds are measured on numpy, not PHP, and the “I cannot answer” rule judges the ABSOLUTE best cosine, which an approximate index may miss. So the exhaustive scan stays as long as this measure fits the budget.
- web_search_helper.build(passages, embedder, lang, stop=None, thresholds=None, limits=None, ranking=None, batch=256)[source]
Cut, embed and index
passages.
- web_search_helper.calibrate(idx, inside, outside, embedder)[source]
New thresholds and a report;
idx.metais left untouched.
- web_search_helper.check_online(base_url, langs, question='test', data_dir='search-data')[source]
Is the corpus private, and does the endpoint answer?
- web_search_helper.from_vectors(rows, vectors, lang, model, stop=None, thresholds=None, limits=None, ranking=None)[source]
Index passages whose windows are already embedded, by another pipeline.
For a site that computes its own embeddings and wants the engine without re-embedding (or wants the SAME vectors, to compare engines).
rowsare dicts witht(text),c(document),s(section),o(end offset of each window int) and optionallyu(url) andm(meta passed to the browser);vectorsholds one L2-normalised row per window, passages in order.modelnames the Python model and its transformers.js twin, which the page must use for the question.>>> rows = [{"t": "Un éléphant.", "c": "D", "s": "S", "o": [12]}] >>> idx = from_vectors(rows, np.eye(1, 4, dtype=np.float32), "fr", ... {"python": "m", "browser": "Xenova/m"}) >>> idx.meta["dim"], idx.words (4, ['elephant'])
- web_search_helper.install(site, indexes, data_dir='search-data', demo=False)[source]
Copy the endpoint, the client and the index files into
site.demoalso copies the bilingual demo page asdemo.html.
- web_search_helper.read_corpus(path, target_words=220)[source]
A
.jsonlfile, a single Markdown/text file, or a folder of them.
- web_search_helper.search(idx, q, qv=None, explain=False)[source]
What
search.phpanswers toq(and the question vectorqv).