feat(brain): hybrid BM25 + pgvector retrieval (opt-in)
All checks were successful
CI / Lint / Test / Vet (push) Successful in 15s
CI / Mirror to GitHub (push) Successful in 3s

Wires nomic-embed-text (iguana ollama) + pgvector on the shared
postgres18 into brain_query / brain_answer via Reciprocal Rank Fusion.
Pure BM25 stays the default; setting BRAIN_PG_DSN and BRAIN_EMBED_URL
together opts in. Setting one without the other is misconfiguration →
exit 1.

New packages:

- internal/embed
  Client.Embed(ctx, text) → []float32 via POST {URL}/api/embed.
  Defaults to nomic-embed-text:latest (768 dim). nil-on-empty-URL so
  callers gate on a single nil check.

- internal/vectorstore
  PGStore wraps a pgxpool against postgres18. Init creates
  brain_embeddings(path PK, vector(768), updated_at) + HNSW cosine
  index idempotently. Upsert / Delete / Search / KnownPaths.
  Sync(brainDir, store, embedder) diffs brain/wiki/ against the store
  and upserts new files / deletes removed ones; StartSync runs it on
  a ticker (default 300s). Integration tests gated by BRAIN_PG_TEST_DSN.

- scripts/brain-embeddings-init.sql
  One-time DBA setup: brain DB, brain_app role, vector extension,
  GRANTs. Idempotent.

Search layer:

- search.QueryOptions gains Vector + Embedder fields.
- QueryContext is the cancellable variant; Query stays for callers.
- When both are set, BM25 (top-N) and pgvector (top-4N) candidates
  merge via Reciprocal Rank Fusion (k=60, Cormack et al. 2009 — no
  tuning knob, robust to scale differences between rankers).
- Vector-only hits are hydrated from disk so callers see uniform
  Result records (path, title, excerpt, wing, hall, score).
- Wing/hall filters still apply to vector candidates via path-prefix.
- On embedder/vector errors the search falls back to BM25 — embedding
  outage degrades quality but doesn't take the brain offline.

MCP wiring:

- mcp.Server.WithHybridRetrieval(v, e) opt-in setter, same shape as
  WithReranker.
- brainQuery and brainAnswer pass the wired vector/embedder through
  to search.QueryContext.

REST:

- POST /backfill-embeddings drives Sync synchronously. Returns
  {added, deleted, errors[]}. 503 when feature is unconfigured.

cmd/server/main.go:

- BRAIN_PG_DSN + BRAIN_EMBED_URL together enable hybrid; one alone
  → exit 1.
- vectorAdapter bridges *PGStore (returns []Hit) to
  search.VectorSearcher (which takes []VectorHit) without either
  package importing the other.
- BRAIN_EMBED_SYNC_INTERVAL (default 300s) controls the background
  Sync ticker.

Backend pivot from Qdrant to pgvector recorded in DECISIONS.md
2026-05-18 (supersedes 2026-04-08): postgres18 already runs in
databases/ ns, Qdrant was never deployed, one engine beats two.

Dependency: github.com/jackc/pgx/v5 — modern, native pgvector via
parametric vector literals.

Tests:
- embed.Client: empty-URL nil, request shape, dimension, upstream
  error propagation, empty-text rejection.
- vectorstore.PGStore: dimension validation (unit); upsert/search/
  KnownPaths (integration, BRAIN_PG_TEST_DSN-gated).
- vectorstore.Sync: adds new files, skips known, deletes
  disappeared, skips _index.md, no-op when nil, collects embedder
  errors.
- search.Query: hybrid promotes vector-only hits via RRF; falls
  back to BM25 on embedder error.

Closes hyperguild#8.
This commit is contained in:
Mathias
2026-05-18 23:11:25 +02:00
parent a56a4db963
commit 57462b52ff
16 changed files with 1068 additions and 14 deletions

View File

@@ -2,6 +2,7 @@
package search_test
import (
"context"
"fmt"
"os"
"path/filepath"
@@ -12,6 +13,69 @@ import (
"github.com/stretchr/testify/require"
)
type stubEmbedder struct{ vec []float32 }
func (s stubEmbedder) Embed(_ context.Context, _ string) ([]float32, error) { return s.vec, nil }
type stubVector struct{ hits []search.VectorHit }
func (s stubVector) Search(_ context.Context, _ []float32, _ int) ([]search.VectorHit, error) {
return s.hits, nil
}
func TestSearch_HybridRRFPromotesVectorOnlyHit(t *testing.T) {
dir := t.TempDir()
for _, p := range []struct{ rel, body string }{
// BM25-keyword note (matches "lejpa" once)
{"wiki/jepa-fx/facts/foo.md", "---\ntitle: Foo\n---\nlejpa keyword\n"},
// Semantically related note that does NOT contain the keyword.
{"wiki/jepa-fx/facts/semantic.md", "---\ntitle: Semantic\n---\nNo keyword in body.\n"},
} {
full := filepath.Join(dir, p.rel)
require.NoError(t, os.MkdirAll(filepath.Dir(full), 0o755))
require.NoError(t, os.WriteFile(full, []byte(p.body), 0o644))
}
embedder := stubEmbedder{vec: []float32{0.1}}
vector := stubVector{hits: []search.VectorHit{
{Path: "wiki/jepa-fx/facts/semantic.md", Distance: 0.05}, // best vector match
{Path: "wiki/jepa-fx/facts/foo.md", Distance: 0.10},
}}
got, err := search.Query(dir, search.QueryOptions{
Query: "lejpa",
Limit: 5,
Vector: vector,
Embedder: embedder,
})
require.NoError(t, err)
require.Len(t, got, 2, "vector-only hit should be hydrated into results")
paths := []string{got[0].Path, got[1].Path}
assert.Contains(t, paths, "wiki/jepa-fx/facts/foo.md")
assert.Contains(t, paths, "wiki/jepa-fx/facts/semantic.md")
}
func TestSearch_HybridFallsBackOnEmbedderError(t *testing.T) {
dir := t.TempDir()
require.NoError(t, os.MkdirAll(filepath.Join(dir, "wiki"), 0o755))
require.NoError(t, os.WriteFile(filepath.Join(dir, "wiki", "x.md"), []byte("keyword foo"), 0o644))
embedder := errorEmbedder{}
vector := stubVector{}
got, err := search.Query(dir, search.QueryOptions{
Query: "keyword", Limit: 5, Vector: vector, Embedder: embedder,
})
require.NoError(t, err)
require.Len(t, got, 1, "BM25 result should still come back when embedder fails")
assert.Equal(t, "wiki/x.md", got[0].Path)
}
type errorEmbedder struct{}
func (errorEmbedder) Embed(_ context.Context, _ string) ([]float32, error) {
return nil, assert.AnError
}
func TestSearch_ReturnsMatchingPages(t *testing.T) {
dir := t.TempDir()
require.NoError(t, os.MkdirAll(filepath.Join(dir, "knowledge"), 0o755))