web_search_helper.calibrate module
Measure the thresholds of the “I cannot answer” rule on YOUR corpus.
The engine answers only when one of three routes says the corpus treats the
question (see engine.sure_enough). Each route has a threshold, and a
threshold guessed is a threshold wrong: the cosines move every time the corpus
is re-indexed. So they are measured, from two lists of questions you write:
inside: questions the corpus answers;outside: questions it does not, ideally some that borrow its vocabulary (they are the hard ones).
Each threshold is placed halfway between the two families, on the questions that this route alone must decide:
df_min: one-word queries; between the most frequent outside word and the rarest inside word;sem_min: between the highest outside best-window cosine and the lowest inside one among the questions the word route does not already save;sec_min: the same, on the best section centroid;lex_min: between the best outside BM25 that passed the conjunctive pass and the lowest inside one.
When the families overlap, no value separates them; the report says so and keeps the previous value. Then the whole rule is replayed, and every question that still falls on the wrong side is listed.
The question vectors here come from the Python model; the page computes them with transformers.js. The two agree to about 1e-2 on cosines, which is why a margin is reported: a threshold with less margin than that is fragile.
- class web_search_helper.calibrate.Measure(q: 'str', inside: 'bool', has_terms: 'bool', typed: 'int', df1: 'int', conj: 'bool', best: 'float', top: 'float', sectop: 'float')[source]
Bases:
object- Parameters: