Wires nomic-embed-text (iguana ollama) + pgvector on the shared
postgres18 into brain_query / brain_answer via Reciprocal Rank Fusion.
Pure BM25 stays the default; setting BRAIN_PG_DSN and BRAIN_EMBED_URL
together opts in. Setting one without the other is misconfiguration →
exit 1.
New packages:
- internal/embed
Client.Embed(ctx, text) → []float32 via POST {URL}/api/embed.
Defaults to nomic-embed-text:latest (768 dim). nil-on-empty-URL so
callers gate on a single nil check.
- internal/vectorstore
PGStore wraps a pgxpool against postgres18. Init creates
brain_embeddings(path PK, vector(768), updated_at) + HNSW cosine
index idempotently. Upsert / Delete / Search / KnownPaths.
Sync(brainDir, store, embedder) diffs brain/wiki/ against the store
and upserts new files / deletes removed ones; StartSync runs it on
a ticker (default 300s). Integration tests gated by BRAIN_PG_TEST_DSN.
- scripts/brain-embeddings-init.sql
One-time DBA setup: brain DB, brain_app role, vector extension,
GRANTs. Idempotent.
Search layer:
- search.QueryOptions gains Vector + Embedder fields.
- QueryContext is the cancellable variant; Query stays for callers.
- When both are set, BM25 (top-N) and pgvector (top-4N) candidates
merge via Reciprocal Rank Fusion (k=60, Cormack et al. 2009 — no
tuning knob, robust to scale differences between rankers).
- Vector-only hits are hydrated from disk so callers see uniform
Result records (path, title, excerpt, wing, hall, score).
- Wing/hall filters still apply to vector candidates via path-prefix.
- On embedder/vector errors the search falls back to BM25 — embedding
outage degrades quality but doesn't take the brain offline.
MCP wiring:
- mcp.Server.WithHybridRetrieval(v, e) opt-in setter, same shape as
WithReranker.
- brainQuery and brainAnswer pass the wired vector/embedder through
to search.QueryContext.
REST:
- POST /backfill-embeddings drives Sync synchronously. Returns
{added, deleted, errors[]}. 503 when feature is unconfigured.
cmd/server/main.go:
- BRAIN_PG_DSN + BRAIN_EMBED_URL together enable hybrid; one alone
→ exit 1.
- vectorAdapter bridges *PGStore (returns []Hit) to
search.VectorSearcher (which takes []VectorHit) without either
package importing the other.
- BRAIN_EMBED_SYNC_INTERVAL (default 300s) controls the background
Sync ticker.
Backend pivot from Qdrant to pgvector recorded in DECISIONS.md
2026-05-18 (supersedes 2026-04-08): postgres18 already runs in
databases/ ns, Qdrant was never deployed, one engine beats two.
Dependency: github.com/jackc/pgx/v5 — modern, native pgvector via
parametric vector literals.
Tests:
- embed.Client: empty-URL nil, request shape, dimension, upstream
error propagation, empty-text rejection.
- vectorstore.PGStore: dimension validation (unit); upsert/search/
KnownPaths (integration, BRAIN_PG_TEST_DSN-gated).
- vectorstore.Sync: adds new files, skips known, deletes
disappeared, skips _index.md, no-op when nil, collects embedder
errors.
- search.Query: hybrid promotes vector-only hits via RRF; falls
back to BM25 on embedder error.
Closes hyperguild#8.
Adds an opt-in cross-encoder rerank step between BM25 retrieval and LLM
synthesis. With BRAIN_RERANKER_URL set, brain_answer retrieves BM25
top-20, scores each excerpt against the query via Qwen3-Reranker on
Ollama, drops the "no" answers, and forwards up to 5 surviving sources
to the LLM. Unset, behaviour is unchanged (BM25 top-10 → LLM).
The reranker is a *filter*, not a re-ranker: Qwen3-Reranker emits a
binary yes/no token under its native chat template, and ties within the
"yes" set are broken by BM25 rank — what got retrieved first stays
ahead.
New package ingestion/internal/reranker:
- Client with URL, Model, HTTP fields.
- New(url, model) returns nil on empty url so callers can treat
"feature disabled" as a single nil check.
- Score(ctx, query, docs) issues one /api/generate call per doc using
the Qwen3-Reranker yes/no chat template (verbatim, because the model
was trained on this exact wording). Parses the first non-think token.
Wiring:
- mcp.Server gains a WithReranker fluent setter to keep NewServer
signature stable.
- brain_answer's BM25 limit jumps to 20 only when a reranker is wired,
to give the filter something to do.
- cmd/server/main.go reads BRAIN_RERANKER_URL (+ optional
BRAIN_RERANKER_MODEL, default dengcao/Qwen3-Reranker-0.6B:F16).
Tests cover: nil-on-empty-url, ordered yes/no scoring, request shape
(model, prompt contents, yes/no template), ambiguous response → 0,
empty doc slice, upstream-error propagation, plus an end-to-end
brain_answer integration that proves only the relevant note reaches the
LLM when noise.md is rejected.
Closes hyperguild#7.
Adds a two-dimensional address (wing, hall) to brain notes. A wing is a
topic domain (e.g. jepa-fx, hyperguild); a hall is one of a closed
vocabulary of memory types (facts, decisions, failures, hypotheses,
sources). Notes route to brain/wiki/<wing>/<hall>/<slug>.md with
wing/hall/created_at YAML frontmatter, making the directory a valid
Obsidian vault.
Changes:
- new package ingestion/internal/brain (NotePath, ValidHalls, Sanitise,
BuildWingIndex, BuildAllWingIndexes)
- api.WriteNote refactored to WriteNoteOptions; wing+hall routes to
brain/wiki/, otherwise falls back to brain/knowledge/ (legacy)
- search.Query → QueryOptions with optional Wing/Hall filtering; Result
carries wing/hall extracted from frontmatter or path segments
- MCP tools brain_write and brain_query gain optional wing/hall params
(hall enum-validated); new brain_index tool regenerates _index.md MOC
- POST /index REST endpoint mirrors brain_index
- brain_write auto-rebuilds the wing's _index.md after a wing+hall write
- scripts/migrate-brain-halls.sh migrates flat brain/wiki/{concepts,entities}/
into the new layout (dry-run by default, --commit applies)
All existing tests pass; new tests cover wing/hall write routing, scope
filtering, invalid hall rejection, _index.md generation, and migration
script paths.
Closes hyperguild#1.
Adds two new LLM-backed MCP tools to the ingestion service:
- brain_answer(query): BM25 retrieval + LLM synthesis → answer + sources
- brain_classify(text): classifies doc into type/title/tags via LLM
Adds llm.Router for primary→fallback routing (berget.ai → iguana).
Wired via BRAIN_LLM_PRIMARY_URL/BRAIN_LLM_FALLBACK_URL env vars;
no-op when unset so existing deployments are unaffected.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
