web_search_helper.corpus module
Read a corpus into passages.
A passage is the unit the engine ranks and returns an excerpt of. It carries
the text, the document and section it comes from (shown as the source), an
optional url the page can link to, and a free meta dict passed through
untouched to the browser.
Two inputs:
JSONL, one passage per line:
{"text": …, "doc": …, "section": …, "url": …, "meta": {…}}. Use it when the passages are already cut, by your own pipeline.A folder of Markdown or text files: each
#heading opens a section, the first heading (or the file name) titles the document, and paragraphs are grouped into passages of abouttarget_wordswords.
>>> ps = passages_from_markdown("# Doc\n\n## Part\n\nOne two three.", "d.md")
>>> [(p.doc, p.section, p.text) for p in ps]
[('Doc', 'Part', 'One two three.')]
- class web_search_helper.corpus.Passage(text: 'str', doc: 'str' = '', section: 'str' = '', url: 'str' = '', meta: 'dict' = <factory>)[source]
Bases:
object
- web_search_helper.corpus.passages_from_markdown(source, name, target_words=220)[source]
Cut one Markdown (or plain text) document into passages.
Paragraphs are never split: a passage closes at the first paragraph boundary past
target_words, and never spans two sections.
- web_search_helper.corpus.read_corpus(path, target_words=220)[source]
A
.jsonlfile, a single Markdown/text file, or a folder of them.