pacod-kb

Why it exists

Most of the work in retrieval-augmented generation happens before a model reads a word: getting documents in, keeping them clean, finding the right passage, and telling the model where it came from. pacod-kb is that part, built as a service of its own.

Three demands shaped it. Answers must be citable, so every chunk carries its heading path and byte offsets into a stable Markdown body. Sources change, so nothing is overwritten: a new version supersedes the old one, which stays searchable by date. And ingested text is untrusted input to a model, so it is scanned for prompt injection before indexing and fenced as data on the way out.

From documents to answers

A document takes the same path whether it was uploaded, dropped into the dropzone folder, fetched from a URL, or re-fetched from a watched web source:

1. Acquire. The SHA-256 of the source bytes is the document id, so the same file is never ingested twice.
2. Parse. PDF (docling, with optional OCR for scanned pages), DOCX, HTML, Markdown, text and JSON become Markdown with YAML front matter.
3. Check the language. English, German, Bulgarian, Russian, Serbian, Macedonian, Croatian and Slovenian are accepted; others are refused.
4. Scan the deterministic parse for prompt injection and spam, before any LLM sees the text.
5. Convert (optional). An LLM pass through pacod-gw tidies the Markdown. A cut-off, refused or empty answer keeps the deterministic parse.
6. Clean and structure. A noise filter strips page numbers, running headers and boilerplate; heading promotion recognises the structure of legal texts, papers and articles.
7. Store document.md, a manifest and a snapshot of the source.
8. Index. Heading-aware chunks of up to 500 tokens are embedded with BGE-M3 into Qdrant and written to a keyword index, in the project's own collection.

Architecture

pacod-kb is two services, kb-pipeline and kb-ui, plus the model and storage they use:

The split is deliberate. The pipeline parses untrusted documents and fetches arbitrary URLs, so it receives none of pacod-id's secrets; the UI never sees the gateway key. All containers join one shared Docker network, pacod-net, so the separation rests on secrets, not networks: each container gets an explicit list of environment variables, and a test checks it.

Every call from the UI carries X-KB-Actor, the signed-in person's id. The pipeline trusts it only together with the API key, records it on jobs, settings changes and usage, and passes it to the gateway. When pacod-id deletes an account, kb replaces that id with deleted-user everywhere before confirming.

Security model

Content security. The scan normalises text to defeat obfuscation, matches a multilingual rule set, and returns one of three verdicts:

• block: a single high-severity injection or obfuscation finding, or a high threat score. The ingest fails and nothing is stored.
• quarantine: the document is kept with status review and never indexed. Spam alone never leads to a block.
• clean: the document is indexed.

An optional LLM second opinion through the gateway is asked only when the rules did not block; it has 15 seconds and fails open. Known-safe material can skip the scan with kb ingest --trusted. On the way out, /query wraps each chunk in an untrusted-data fence.

Fetching. WEB_FETCH_ALLOW_PRIVATE=false, recommended in production, refuses URL ingests that resolve to private addresses: a guard against server-side request forgery.

People. The UI signs people in through pacod-id with OIDC and PKCE and keeps the tokens in an encrypted server-side session; the cookie carries only a random secret. Roles come from pacod-id's authorization feed. If the UI's copy is more than 60 seconds old, pages answer 503 rather than guess who may do what.

Search, measured

A query runs against Qdrant and the keyword index, and the two rankings are fused with reciprocal rank fusion (k = 60). Date and history filters apply inside both searches. An optional cross-encoder, BAAI/bge-reranker-v2-m3, reorders the best candidates and gives a 0–1 relevance score that is comparable across queries, so min_score can drop non-answers.

The defaults come from an evaluation harness: 14 documents in six languages (GDPR, the Law Enforcement Directive, four constitutions, Wikipedia articles) and 92 hand-written queries — paraphrases, exact terms, cross-lingual, native-language and unanswerable. Measured on a development machine's CPU, which the docs stress are not production numbers:

• Without the reranker, the right section is in the top five for 98.7 % of answerable queries; R@1 is 0.83, at about 50 ms.
• With the reranker, R@1 rises to 0.91, at about 2 s per query.
• Under heavy ingest load, a search used to wait about 24 s for a whole embedding batch. Queries now go ahead of ingest work and wait for one small micro-batch at most: about 0.4 s (p50) with ingest jobs run one after another, as the pipeline runs them.
• Length-sorted batches make ingest embedding about 2.2× faster, with unchanged vectors.

The reranker is off by default: its gain fell just short of the bar on the held-out queries, and its latency has to be confirmed on the production CPU.

What a model receives

POST /query returns chunks under a stable, documented metadata contract:

{"query": "maximum fine for a data breach", "top_k": 5, "as_of": "2025-12-30"}

The response is {"results": [...], "reranked": false, "score_kind": "rrf"}. Each result carries:

Each document also has a manifest: title, source, document type, status, project, the parser or model that produced the body, and a lifecycle log (ingested, superseded, security_flagged, llm_usage and more).

Versions, not overwrites

A new version ingested with --supersedes, or a watched source that has changed, marks the old one with valid_until — in the front matter, the manifest, the keyword index and the vector payloads at once. Searches return current chunks by default; as_of returns what was valid on a given day; include_historical returns everything. If a watched page reverts, the earlier version is reinstated. Supersessions and rebuilds reuse the vectors of unchanged chunks.

Running it

kb follows the pacod shared-network convention: every container joins one Docker network, pacod-net. docker-compose.yml holds kb's own services, kb-pipeline and kb-ui. docker-compose.standalone.yml holds kb's copies of the shared components, switched on by Compose profiles: Qdrant, PostgreSQL, pacod-id with its database next to kb's, a mail catcher, and a development gateway whose mock providers keep everything on the machine. make up loads both files. Only one copy of each shared component can run on a machine. In production you point kb at a real pacod-gw, put pacod-id behind TLS, and drop the development profiles.

pacod-demo, live at demo.pacod.io, runs kb together with the other pacod projects. It loads only kb's app file: kb keeps its database on the shared PostgreSQL and its collections on the shared Qdrant, embeds through the shared embeddings service, and serves its UI behind the shared Caddy edge.

• Storage. KB_STORAGE_BACKEND is filesystem or database; switching does not migrate documents.
• Data. The data directories are variables (KB_ROOT, KB_MODEL_CACHE, KB_QDRANT_DATA, KB_POSTGRES_DATA). A name without a slash is a Docker volume; make setup uses volumes on macOS.
• CPU. TORCH_NUM_THREADS=0 follows the container's CPU quota. EMBED_BATCH_TOKENS (1024) bounds how long a query can wait behind an ingest.
• Offline models. Run once with network access to fill the model cache, then set HF_HUB_OFFLINE=1.
• Shared models. EMBEDDINGS_URL embeds through the shared embeddings service instead of loading BGE-M3. The vectors are the same, so nothing is re-embedded, and the pipeline needs about 2 GB less memory. RERANKER_URL does the same for the reranker.
• Shared Qdrant. On a Qdrant that other products use, the UI's Qdrant pages show and touch only kb's collections (KB_QDRANT_COLLECTION_PREFIX, default kb_).
• Runtime switches. The dashboard turns the content scan, AI conversion and language validation on and off without a restart; environment variables win at the next start.
• AI. Admins choose models and edit prompts in AI Management. Every call carries opaque attribution (feature, job, project, person), and kb records the tokens with the gateway's cost. With no gateway configured, AI is off and nothing else breaks.

Backups cover KB_ROOT or kb's PostgreSQL database, the Qdrant storage (or rebuild it with kb rebuild --all-projects), pacod-id's database on the same PostgreSQL with its key-encryption key, and .env.

Limits and further reading

• The pipeline and the UI each run as a single worker: embedded Qdrant holds a file lock, and the UI's session handling assumes one process.
• A reranker in the pipeline's own process roughly doubles its memory; with RERANKER_URL the pipeline loads none.
• Serbian is accepted, but the language detector has no Serbian profile, so Serbian text may be misdetected or rejected.
• Documented for the next reindex, not shipped: BGE-M3 sparse vectors for the keyword half, chunker clean-ups, and rebuilding into a shadow collection before swapping.

To go deeper:

• Architecture: containers, sign-in, the gateway, who acted.
• Metadata contract: every chunk and manifest field.
• Retrieval evaluation: the full numbers and the decisions behind the defaults.
• Runbook: development, production, operations and troubleshooting.
• Migrations: upgrading an existing installation, including the move to pacod-net.