DEV Community: fu jun

From a Pile of Office Files to a Searchable Local Knowledge Base: I Built an Open-Source Skill

fu jun — Mon, 02 Nov 2026 02:57:52 +0000

From a Pile of Office Files to a Searchable Local Knowledge Base: I Built an Open-Source Skill

Self-hosted knowledge base skill v4.0 is out: https://github.com/furoute/workshop/tree/main/skills/self-hosted-kb
Companion skill: Everything file search: https://github.com/furoute/workshop/tree/main/skills/everything-http-search

The problem

I had tens of GB of Office/PDF files sitting around: proposals, contracts, notices, monthly reports. When I needed something, all I had was filenames and memory. Even a well-named file can't answer "what was the winning bid for that project?" or "how does XX's proposal handle operations?"

Off-the-shelf knowledge base options either live in the cloud (your data leaves home) or require standing up a stack of services. What I wanted was small: files stay on my own disk, I can find the original text and open the original file, and when I ask a question the model answers with sources. That tiny list of requirements meant building the pipeline myself.

So I made a skill: self-hosted-kb, a complete hands-on record, MIT licensed. Not a product — a workbook you can follow step by step and get running.

The pipeline: seven steps + two retrieval paths

One sentence: files stay in place, read-only; the index and vectors live separately; at query time, two paths retrieve and fuse.

Seven steps (read-only first, changes later):

Full probe (read-only): inventory + magic-number sniffing, four-way triage (healthy / format-faked / damaged / encrypted-or-locked)
SHA-256 dedup (read-only): exact duplicates only count by hash — same name and size doesn't count
Stratified sample reads (read-only): sample 400–500 files by directory × year × size; recompute all SHAs after the reads and require zero changes — proof I didn't touch the files
Auto-triage into three piles (labels only): useful / junk / undecided; undecided files get grouped into "file families", because nobody can review tens of thousands of rows
Batch adjudication (labels only): 28 rules, one final state per family
Entity cards: five entity types, fifteen YAML fields + nine body sections; evidence grades never get inflated, money figures use a fixed convention
Presentation layer (optional): index + property views + Q&A pages in Obsidian, for humans

Retrieval stands on two legs:

FTS5 (keywords): model numbers, IDs, amounts, dates — exact literal matches
Vectors (semantics): fuzzy questions like "how do I handle this"
RRF rank fusion: what ranks high on both paths ranks highest

Scale from the real run: the largest batch was 37,782 files / 27.7 GB → after inventory and dedup, 2,066 files / 10.7 GB; 400k chunks, a 950 MB index.

Tuning: the data was in the base, the model said it wasn't

The most valuable part. Once, a user asked about something that was definitely in the base, and the model answered "not found" — but a direct FTS5 query returned 10 precise hits. The data wasn't missing; it was buried. Two root causes:

1. FTS5's default trigram tokenizer kills two-character Chinese words. Trigrams index 3-character sliding windows, so two-character Chinese words like "合同" (contract) or "预算" (budget) can never form a complete trigram — in testing, 11 two-character queries returned 0 hits, all 11 of them. The fix: jieba segmentation + unicode61, tokenize before feeding FTS5. Lesson: on Chinese corpora, the tokenization strategy matters an order of magnitude more than the index structure.

2. One giant Excel outweighed 130 PDFs. One spreadsheet's extracted text cut into 50,000 chunks, while the library-wide average was just 365 words per chunk — no matter the question, that spreadsheet's chunks filled the candidate pool and pushed the right documents down. That's "retrieval pollution", not "retrieval inaccuracy": the fix isn't better scoring, it's stopping big documents from hogging the pool. Five fixes: chunk-count downweighting, table-density downweighting, a cap of 20 chunks per document, directory-path evidence, and cutting query terms from 60 to 12.

Before vs after: latency 96–203 ms → 24–96 ms (it got faster, because the candidate pool shrank from thousands to hundreds); three groups of previously unanswerable questions all hit the right documents.

The mindset in one line: don't "score the right answer higher" — "keep the wrong answers out of the candidate pool".

Everything: finding files is a different layer, don't mix it with retrieval

"Does file X exist?" needs no vectorization at all — Everything reads the NTFS master file table and locates files among hundreds of thousands in milliseconds, zero tokens. The order should be:

"Does X exist?" → Everything (milliseconds, zero cost)
"What does X say?" → knowledge base retrieval (chunking + vectors)

80% of "find a file" scenarios need no vectorization at all. The companion skill is published separately: it runs over an HTTP API, one curl line to query; set up autostart + silent tray + always-on HTTP once. Note: this is a Windows + NTFS solution; on Linux/macOS, use grep/find.

Cloud vs local models: the bottleneck is retrieval, not the model

Same retrieval chain, two answering models: cloud DeepSeek API vs local Ollama qwen3.5:9b (RTX 4060 Laptop, network cable unplugged throughout).

Speed: cloud steady at 0.7–1.1 s; local jittery at 0.9–4.1 s
Quality: understanding, summarizing, reading proposals — roughly tied; but the local small model has two hard flaws: multi-digit multiplication goes wrong, and tables squeezed into one line get their states mixed up; when information is insufficient, the cloud model leans toward "not sure", the local one leans toward confabulating
Key finding: questions where retrieval handed over the wrong material were unanswerable on both sides — a bigger model can't conjure a document that never entered the candidate pool. Optimize recall quality, not model size
One hands-on pitfall: qwen3.5 has thinking on by default, and the chain of thought eats the whole num_predict budget, leaving an empty answer — you have to turn it off explicitly

Data boundary (honest version)

The accurate statement: data doesn't leave home in bulk, but the snippets hit at query time — and your question — do go out.

Stays home: the original corpus, 400k chunks, the 950 MB index, the vectors — all on my own disk
Goes out: the top-k snippets selected at query time + the user's question, sent to the cloud model for understanding and organization
Looking up a technical manual: fine. Looking up a client list, a quote, contract terms: those passages do go out — so judge before putting sensitive material in the base
The only way to keep everything home is a local model (5–10× slower, one tier weaker in Chinese)

How to use it

This isn't SaaS; there's no one-click installer. The right way: hand the skill folder to your agent and let it follow SKILL.md:

https://github.com/furoute/workshop/tree/main/skills/self-hosted-kb

SKILL.md is a workbook written for an agent to follow, not a manual for humans; companion scripts, references and architecture diagrams are in the same directory. The Everything skill is here:

https://github.com/furoute/workshop/tree/main/skills/everything-http-search

Honest boundaries

One complete hands-on record, not a general product; a different corpus means re-tuning
Windows + NTFS is the precondition for the Everything layer; legacy Office formats (.doc/.xls/.ppt) aren't parsed by the current scripts — only their filenames are searchable
Hands-on ability is a precondition: you run the scripts, build the index, and debug it yourself
Example data is de-identified; replace personal configuration with your own

Closing

The original intent was simple: my files, my retrieval. However convenient a cloud knowledge base is, once data leaves, it's gone; once this local chain works, finding it, opening it, and citing the source is enough.

Welcome to try it — issues go straight to the repo.

Side note: this wasn't a solo effort — it was built by six AI agents across three machines, with task split, review and merge all running over ACP-lite (Agent Connect Personal), an open-source file-mailbox-style interconnection scheme: https://github.com/furoute/acp-lite

How Do AI Agents on Different Computers Talk to Each Other? I Built a Tiny Open-Source Fix

fu jun — Fri, 16 Oct 2026 06:16:05 +0000

Update (Nov 2026): v0.7.1 is out — delivery facts are now derived from the signed envelope, tickets are bound to (msg_id, requires, data), and 0.7.0 tickets need re-signing. The SSH doorbell, mailbox design and pitfalls below are unchanged. Related new work from the same author: a self-hosted knowledge-base skill (FTS5 + vectors) and an Everything file-search skill: https://github.com/furoute/workshop

How Do AI Agents on Different Computers Talk to Each Other? I Built a Tiny Open-Source Fix

ACP-lite (Agent Connect Personal) v0.7.0 is out: https://github.com/furoute/acp-lite

Note: unrelated to Zed's Agent Client Protocol and OpenAI+Stripe's Agentic Commerce Protocol — same acronym, different project.

The problem

I run 6 AI agents across 3 machines: Codex and DSH on my desktop, Hermes and OpenClaw on a netbook. Human-to-agent is easy — WeChat, DingTalk, done. But how do agents talk to each other?

Search for answers and you get enterprise stacks: Google's A2A, RabbitMQ, Kafka. All built for data centers. My need is much smaller: two home computers, both behind NAT, no public IP, no server — agents passing messages and tasks to each other. Nobody builds for that.

So I built ACP-lite (Agent Connect Personal): a minimal implementation of cross-host agent interconnect. v0.7.0 just shipped, MIT licensed.

The architecture: one post office per machine

The whole idea in one sentence: each machine runs a "post office": agents only ever read and write local files.

Send: agent A writes a letter (JSON) to its local outbox
Transport: Syncthing syncs the file to the other machine — battle-tested decentralized file sync, no need to reinvent it
Receive: agent B picks it up from its local inbox
Trust: Ed25519-signed envelopes + out-of-band fingerprint verification + a hash-chained audit log. Tampered letters are rejected outright.

No central server, no persistent network connections. The watch process only moves files — it never calls a model, so idling costs zero tokens. That matters for personal use: run it for a month, pay nothing extra.

How it relates to the big names: A2A is the enterprise-grade full protocol, ACP-lite is the personal minimal implementation — different leagues. MCP connects agents to tools; ACP-lite connects agents to agents.

New in v0.7.0: cross-host wake-up (the SSH doorbell)

A mailbox alone isn't enough: the letter arrives, but the agent on the other side is asleep. v0.7.0 adds the "doorbell" — SSH wake-up.

The architecture rule is fixed: SSH is the doorbell, ACP is the mailbox; the doorbell carries only a pointer, the letter body always travels via ACP.

The wake-up command passes just the absolute path of the received letter file; the prompt goes through a file + stdin (measured ~20x fewer tokens than passing it as a CLI argument, and no shell-mangled encoding).

Last week's dual-channel integration test, timeline included:

Desktop rebooted: sshd / Syncthing / watch all came back up with zero manual intervention, IP unchanged
Netbook delivered 2 letters; 18 seconds later the doorbell rang; both wake scripts pulled up Codex and DSH within a second
Codex first moved in 6 seconds, DSH in 9; each wrote artifacts + broadcast + signed replies; the audit chain verified end-to-end

The completion criteria are threefold, all required: artifact on disk, longer audit chain, signed receipt. Don't trust exit code 0 alone — an agent woken in read-only sandbox mode will happily "talk about doing the work" and exit 0 with nothing done.

Pitfalls worth knowing (four favorites)

All earned the hard way; the README has the full list (Q5-Q13). Four highlights:

ALL_PROXY hijacks everything.
- Since Syncthing 2.1, it honors only ALL_PROXY and ignores NO_PROXY — a local Clash setup silently captured all dials and everything looked correct but never connected. Debugging order for a new machine: check proxy env vars first, suspect the firewall second.
Never pass non-ASCII text as CLI arguments. Windows shells read UTF-8 as GBK and you get mojibake. Prompts always go through a file + stdin.
Don't omit --root. Omit it and the exit code is still 0 — looking like success — while the letter lands in the machine-private directory where nobody else can see it. The criterion is always "a file really exists in outbox/", never the exit code.
The wake-up sandbox level decides whether it works or performs. Codex wake-ups need workspace-write; DSH wake-ups need DSH_PERMISSION_MODE=danger-full-access baked into the script. Wrong level, and the agent performs diligence theater.

How to use it

Be clear about this: it's not SaaS, there's no one-click installer. The one correct usage — hand the repo link to your coding agent and let it install from the README.

The README is step-by-step: step one do this, step two do that, each step only "what to do + commands to copy verbatim". After reading it, an agent can set the same thing up on your own two machines. There's also an llms.txt if you'd rather feed the whole thing to an LLM at once.

Honest boundaries

A 0-star personal project, only recently made public — don't expect plug-and-play
You'll configure Syncthing and SSH yourself; hands-on ability is a prerequisite.
Built for the personal multi-machine scenario; for enterprise needs, use A2A directly

Why I built this

The motivation is simple: big-vendor solutions are all built for enterprises, nobody builds cross-machine agent interconnect for individual users. ACP-lite is one minimal answer for the "a few of my own computers, a few of my own agents" scenario.

Try it out — issues go directly to the repo. For this project, the first issue from a stranger matters more than a hundred stars.