Skip to content
A document knowledge base that answers two ways
All projects

Software & Data

2026 · 6 tools

A document knowledge base that answers two ways

764 documents, two engines: exact passages from a vector index, or a synthesised overview built from a generated wiki.

RAGChromaDBOpenKBChainlitMCP

scroll ↓

01 / the problem

A working archive of hundreds of files, and no reliable way to ask it a question.

  • 764 files: 522 PDFs, 163 Word documents, 31 spreadsheets, plus email exports
  • Many questions span documents — “what changed between revisions” defeats keyword search
  • An answer is only useful if it can be traced back to a file and a page
  • It has to run on one laptop, offline for retrieval, at no per-query cost

02 / approach

Design choices

01

Two engines instead of one: passages for precision, a wiki for synthesis

02

Local embeddings, so retrieval costs nothing per question

03

Content hashing, so updates only reprocess what actually changed

04

One shared module, so the chat UI and the agent follow identical rules

03 / workflow

How it runs in the field

01

Extract text from PDF, Word, Excel and email exports

02

Chunk with file and page metadata, then embed into a local vector store

03

Build a linked wiki from the same documents for overview answers

04

Expose both behind one chat UI with a Precise / Overview / Both switch

05

Publish retrieval as an agent tool so the same knowledge is queryable programmatically

comes outLocal chat interfaceAgent-facing retrieval toolGenerated wiki (3,829 pages)

04 / implementation

How it was built

architecture · tap any stage

One archive, two engines, one rule set.

Ingest

Extract · chunk · hash

pypdf · python-docx · openpyxl

  • Text is pulled page by page, so a citation can point at a page, not just a file.
  • Chunks are roughly 500 tokens and always keep their file name and page number as metadata.
  • Each file is hashed, so a re-run only re-embeds what actually changed — seconds, not a rebuild.

local only · nothing leaves the machine

02 / 05

01

Ingest

pypdf / python-docx / openpyxl

Text is pulled page by page from PDFs, Word files and spreadsheets, then split into roughly 500-token chunks with the source file and page number kept as metadata.

02

Retrieval

bge-small + ChromaDB

Each chunk is embedded locally into a 384-dimension vector store (288 MB). A question returns the closest chunks, each labelled with its file and chunk number.

03

Synthesis

OpenKB wiki

The same documents are processed a second time into a linked wiki of summaries, concepts and entities — the layer that can answer a question spanning twenty files.

04

Sync

content hashing

Every file is hashed. Re-running the updater re-embeds only changed files and adds only new ones to the wiki, so keeping it current is cheap.

05

Interfaces

Chainlit + MCP

One chat UI with three answer modes, and one agent tool exposing the same retrieval — both importing a single engine module so the rules cannot drift.

results

764
Documents indexed

PDF · Word · Excel

3,829
Wiki pages

generated

288 MB
Vector index

fully local

3
Answer modes

precise · overview · both

05 / tools & stack

What it was built with

Ingestion

pypdf · python-docx · openpyxl
Text extraction across 522 PDFs, 163 Word files and 31 spreadsheets.
~500-token chunking
Chunks carry file name and page number, which is what citations are built from.
Content hashing
Decides what to re-embed, so updates take seconds instead of a full rebuild.

Retrieval

BAAI/bge-small-en-v1.5
Local 384-dimension embeddings — no API calls, no per-query cost.
ChromaDB
Local vector store (288 MB) holding the chunk index.
Top-k with metadata
Returns the nearest chunks with their source label, so every claim can be traced.

Synthesis

OpenKB
Builds the linked wiki — summaries, concepts and entities across the whole archive.
3,829 wiki pages
The layer that answers overview questions no single chunk contains.

Interfaces

Chainlit
Chat UI with Precise / Overview / Both modes, web search and PDF export.
MCP server
Exposes search and status as tools, so an agent can query the same knowledge base.
Shared engine module
One file defines retrieval and answer rules for both front-ends.

Runs entirely on one laptop: local embeddings, local vector store, no documents leave the machine. The archive itself is not published — only the architecture and the numbers above.

next project

Mobile LiDAR grave survey prototype