web_search_helper.corpus module

Read a corpus into passages.

A passage is the unit the engine ranks and returns an excerpt of. It carries the text, the document and section it comes from (shown as the source), an optional url the page can link to, and a free meta dict passed through untouched to the browser.

Two inputs:

  • JSONL, one passage per line: {"text": …, "doc": …, "section": …, "url": …, "meta": {…}}. Use it when the passages are already cut, by your own pipeline.

  • A folder of Markdown or text files: each # heading opens a section, the first heading (or the file name) titles the document, and paragraphs are grouped into passages of about target_words words.

>>> ps = passages_from_markdown("# Doc\n\n## Part\n\nOne two three.", "d.md")
>>> [(p.doc, p.section, p.text) for p in ps]
[('Doc', 'Part', 'One two three.')]
class web_search_helper.corpus.Passage(text: 'str', doc: 'str' = '', section: 'str' = '', url: 'str' = '', meta: 'dict' = <factory>)[source]

Bases: object

Parameters:
doc: str = ''
meta: dict
section: str = ''
text: str
to_dict()[source]
Return type:

dict

url: str = ''
web_search_helper.corpus.passages_from_markdown(source, name, target_words=220)[source]

Cut one Markdown (or plain text) document into passages.

Paragraphs are never split: a passage closes at the first paragraph boundary past target_words, and never spans two sections.

Parameters:
Return type:

list[Passage]

web_search_helper.corpus.read_corpus(path, target_words=220)[source]

A .jsonl file, a single Markdown/text file, or a folder of them.

Parameters:
Return type:

list[Passage]

web_search_helper.corpus.read_folder(folder, target_words=220)[source]

Every .md, .markdown and .txt file under folder, sorted by path.

Parameters:
Return type:

list[Passage]

web_search_helper.corpus.read_jsonl(path)[source]
Parameters:

path (str | Path)

Return type:

list[Passage]