feat(vectorstore): re-embed on file mtime > store updated_at (#23)
Removes the TODO in Sync that left files static after their first embed. Edits to brain/wiki/ and brain/knowledge/ now surface in subsequent syncs without manual /backfill-embeddings calls. Approach - Store interface: KnownPaths → KnownPathsWithTime returning path → updated_at. Callers compare against file mtime to detect edits. - PGStore: SELECT path, updated_at FROM brain_embeddings. - Sync groups known chunks by parent path and tracks the EARLIEST updated_at per parent. A file is stale when its mtime is after that oldest chunk's timestamp — any chunk older than the file means at least one chunk hasn't been refreshed since the last edit. - Stale-path rewrite: delete every old chunk for the parent (handles "file shrunk → fewer chunks → orphan rows at higher #NNNN" cleanly), then re-chunk + re-embed + re-upsert. Tests - New: TestSync_ReembedsFileWhenMtimeNewer — file mtime forced into the future vs store updated_at; Sync deletes old chunk + upserts fresh one. - New: TestSync_SkipsFileWhenMtimeOlder — file mtime backdated; Sync is a no-op (no upserts, no deletes). - Updated: stubStore.known is now map[string]time.Time. A zero value resolves to a far-future sentinel so existing "skip if already known" tests keep passing without per-test setup. - pg_test renamed KnownPaths integration → KnownPathsWithTime; asserts updated_at is non-zero and within 5s of insert wall-clock. Backward compat - brain_embeddings rows pre-dating this change carry valid updated_at values (column was always populated via `DEFAULT now()` + ON CONFLICT `updated_at = now()`). No migration needed. Live pod will start re-embedding any file whose source has been edited since its chunks were originally written. Closes gitea/mathias/hyperguild#23.
This commit is contained in:
@@ -8,6 +8,7 @@ import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/jackc/pgx/v5"
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
@@ -120,21 +121,26 @@ func (s *PGStore) Search(ctx context.Context, query []float32, limit int) ([]Hit
|
||||
return hits, nil
|
||||
}
|
||||
|
||||
// KnownPaths returns the path set already present in the store. Used by
|
||||
// the watcher to diff against the wiki/ tree and decide what to upsert.
|
||||
func (s *PGStore) KnownPaths(ctx context.Context) (map[string]struct{}, error) {
|
||||
rows, err := s.pool.Query(ctx, `SELECT path FROM brain_embeddings`)
|
||||
// KnownPathsWithTime returns every embedded chunk path paired with the
|
||||
// row's updated_at. Sync uses the timestamps to decide whether a file
|
||||
// has been edited since its chunks were last embedded — when the file's
|
||||
// mtime exceeds the oldest chunk's updated_at, the file is re-embedded.
|
||||
func (s *PGStore) KnownPathsWithTime(ctx context.Context) (map[string]time.Time, error) {
|
||||
rows, err := s.pool.Query(ctx, `SELECT path, updated_at FROM brain_embeddings`)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("query paths: %w", err)
|
||||
}
|
||||
defer rows.Close()
|
||||
out := make(map[string]struct{})
|
||||
out := make(map[string]time.Time)
|
||||
for rows.Next() {
|
||||
var p string
|
||||
if err := rows.Scan(&p); err != nil {
|
||||
var (
|
||||
p string
|
||||
t time.Time
|
||||
)
|
||||
if err := rows.Scan(&p, &t); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out[p] = struct{}{}
|
||||
out[p] = t
|
||||
}
|
||||
return out, rows.Err()
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user