mathias/hyperguild
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

feat: structured brain — Hall taxonomy, Obsidian-compatible layout, and filtered query #1

New Issue
Closed
opened 2026-05-07 15:02:19 +00:00 by gitea-mcp-bot · 1 comment
gitea-mcp-bot commented 2026-05-07 15:02:19 +00:00
Copy Link

Context

The brain currently stores notes as a flat list of .md files under brain/knowledge/ and brain/wiki/ with a shallow three-way taxonomy (concepts/, entities/, sources/). Retrieval is term-frequency full-text search across all files.

Two problems emerge as the brain grows:

  1. Retrieval precision degrades — "what decisions were made about jepa-fx?" and "what facts do I know about jepa-fx?" return the same result set. There is no way to query by memory type.
  2. Human curation is hard — the flat layout does not map naturally to an Obsidian vault. A human auditing the brain has no navigational structure, no backlinks between related notes across domains, and no clear entry point.

This issue introduces a Hall dimension borrowed from the MemPalace architecture, restructures the wiki directory layout to be natively Obsidian-compatible, and adds a hall filter to brain_query.

Design

Concept: Wings and Halls

Every note lives at a two-dimensional address:

  • Wing — the topic domain (e.g. jepa-fx, hyperguild, bathroom-plumbing). Maps directly to a subdirectory.
  • Hall — the memory type within any Wing. Fixed vocabulary:
Hall What goes here
facts Established truths, reference data, definitions
decisions Architectural choices, tradeoffs accepted, options rejected
failures Bugs, wrong assumptions, dead ends — and why they failed
hypotheses Unverified ideas staged for future validation
sources Ingested external references (replaces wiki/sources/)

Directory layout (after migration)

brain/
├── wiki/
│   ├── jepa-fx/
│   │   ├── facts/
│   │   │   └── lejpa-architecture.md
│   │   ├── decisions/
│   │   │   └── val-vol-r2-as-optimization-metric.md
│   │   └── failures/
│   │       └── ts-jepa-discarded.md
│   ├── hyperguild/
│   │   ├── decisions/
│   │   │   └── routing-pod-local-floor-0.9.md
│   │   └── facts/
│   │       └── tier-detection-logic.md
│   └── _index.md        ← Obsidian vault entry point (auto-generated)
├── knowledge/           ← unchanged: staging area for human review
├── raw/                 ← unchanged: retrospective output
└── sessions/            ← unchanged: JSONL logs

This layout is a valid Obsidian vault. Open brain/ as a vault and you get Wing-level folders in the file explorer, Hall subfolders, and Obsidian's graph view will show natural clusters by Wing.

Obsidian integration properties

  • Every note uses YAML frontmatter with wing, hall, created_at, and optional tags
  • Wikilinks ([[other-note]]) written by retrospective or brain_write will resolve natively in Obsidian
  • _index.md files at Wing level act as MOC (Map of Content) — auto-generated, listing all notes in that Wing with their Hall
  • No Obsidian plugins required — the layout works with vanilla Obsidian

Implementation

1. New path resolver: ingestion/internal/brain/path.go

package brain

// ValidHalls is the closed vocabulary of hall names.
var ValidHalls = map[string]bool{
    "facts": true, "decisions": true, "failures": true,
    "hypotheses": true, "sources": true,
}

// NotePath resolves a canonical filesystem path for a note given wing, hall,
// and slug. Returns error if hall is not in ValidHalls.
func NotePath(brainDir, wing, hall, slug string) (string, error)

Wing and slug are sanitised (lowercase, alphanumeric + hyphens, no path separators). Hall is validated against ValidHalls.

2. Extend WriteNote in ingestion/internal/api/handler.go

Add optional wing and hall fields to writeRequest:

type writeRequest struct {
    Content  string `json:"content"`
    Filename string `json:"filename,omitempty"`
    Type     string `json:"type,omitempty"`   // kept for backwards compat
    Domain   string `json:"domain,omitempty"` // kept for backwards compat
    Wing     string `json:"wing,omitempty"`
    Hall     string `json:"hall,omitempty"`
}

When wing + hall are provided, route to brain/wiki/<wing>/<hall>/<filename> via brain.NotePath. When absent, fall back to current brain/knowledge/ behaviour. This is fully backwards-compatible — no existing callers break.

Inject YAML frontmatter automatically:

---
wing: jepa-fx
hall: decisions
created_at: 2026-05-07T10:00:00Z
---

3. Extend search.Query in ingestion/internal/search/search.go

Add hall and wing filter parameters:

type QueryOptions struct {
    Query string
    Limit int
    Wing  string // optional: restrict to brain/wiki/<wing>/
    Hall  string // optional: restrict to brain/wiki/<wing>/<hall>/
}

func Query(brainDir string, opts QueryOptions) ([]Result, error)

Path-based filtering: when wing is set, only walk brain/wiki/<wing>/. When hall is additionally set, only walk brain/wiki/<wing>/<hall>/. This is O(1) directory scoping — no change to the scoring logic.

Update Result to include wing and hall fields extracted from frontmatter or path segments.

4. Update MCP tool schemas

In ingestion/internal/mcp/:

brain_write — add optional wing (string) and hall (enum: facts/decisions/failures/hypotheses/sources) parameters.

brain_query — add optional wing (string) and hall (enum) parameters.

5. Index generation: brain_index

New MCP tool (and REST endpoint POST /index) that walks brain/wiki/ and regenerates _index.md at each Wing directory:

# jepa-fx

| Hall | Note | Created |
|------|------|---------|
| decisions | [val-vol-r2-as-optimization-metric](decisions/val-vol-r2-as-optimization-metric.md) | 2026-05-06 |
| facts | [lejpa-architecture](facts/lejpa-architecture.md) | 2026-05-04 |

Called automatically after every brain_write with wing+hall, and available as a manual trigger.

6. Migration script: scripts/migrate-brain-halls.sh

One-time script to migrate existing brain/wiki/concepts/ and brain/wiki/entities/ files into the new layout. Reads existing frontmatter type: and domain: fields to infer Wing and Hall. Falls back to wing=general, hall=facts for unmapped files. Non-destructive: leaves originals in place until --commit flag is passed.

Acceptance criteria

  • POST /write with {"wing":"jepa-fx","hall":"decisions","content":"...","filename":"my-decision"} writes to brain/wiki/jepa-fx/decisions/my-decision.md with correct frontmatter
  • POST /write without wing/hall continues to write to brain/knowledge/ unchanged
  • POST /query with {"query":"lejpa","wing":"jepa-fx","hall":"facts"} only searches brain/wiki/jepa-fx/facts/
  • brain_write MCP tool accepts and passes through wing and hall
  • brain_query MCP tool accepts and passes through wing and hall
  • Opening brain/ as an Obsidian vault shows Wing folders in the file explorer with Hall subfolders
  • _index.md at each Wing directory lists all notes in that Wing with Hall column
  • Migration script runs without error on an empty brain and on the existing brain fixture
  • All existing tests pass (backwards-compatible path)
  • New tests cover: valid wing+hall write, invalid hall rejected, wing-scoped query returns only wing results

Branch

feat/brain-halls from main

Out of scope

  • Vector/embedding search (separate issue)
  • Tunnel links between Wings (follow-up after Halls are stable)
  • Obsidian plugin development
  • Changing brain/sessions/ or brain/raw/ layout

Created via git-mcp on behalf of @mathiasbq

## Context The brain currently stores notes as a flat list of `.md` files under `brain/knowledge/` and `brain/wiki/` with a shallow three-way taxonomy (`concepts/`, `entities/`, `sources/`). Retrieval is term-frequency full-text search across all files. Two problems emerge as the brain grows: 1. **Retrieval precision degrades** — "what decisions were made about jepa-fx?" and "what facts do I know about jepa-fx?" return the same result set. There is no way to query by memory type. 2. **Human curation is hard** — the flat layout does not map naturally to an Obsidian vault. A human auditing the brain has no navigational structure, no backlinks between related notes across domains, and no clear entry point. This issue introduces a **Hall dimension** borrowed from the MemPalace architecture, restructures the wiki directory layout to be natively Obsidian-compatible, and adds a `hall` filter to `brain_query`. --- ## Design ### Concept: Wings and Halls Every note lives at a two-dimensional address: - **Wing** — the topic domain (e.g. `jepa-fx`, `hyperguild`, `bathroom-plumbing`). Maps directly to a subdirectory. - **Hall** — the memory type within any Wing. Fixed vocabulary: | Hall | What goes here | |------|---------------| | `facts` | Established truths, reference data, definitions | | `decisions` | Architectural choices, tradeoffs accepted, options rejected | | `failures` | Bugs, wrong assumptions, dead ends — and why they failed | | `hypotheses` | Unverified ideas staged for future validation | | `sources` | Ingested external references (replaces `wiki/sources/`) | ### Directory layout (after migration) ``` brain/ ├── wiki/ │ ├── jepa-fx/ │ │ ├── facts/ │ │ │ └── lejpa-architecture.md │ │ ├── decisions/ │ │ │ └── val-vol-r2-as-optimization-metric.md │ │ └── failures/ │ │ └── ts-jepa-discarded.md │ ├── hyperguild/ │ │ ├── decisions/ │ │ │ └── routing-pod-local-floor-0.9.md │ │ └── facts/ │ │ └── tier-detection-logic.md │ └── _index.md ← Obsidian vault entry point (auto-generated) ├── knowledge/ ← unchanged: staging area for human review ├── raw/ ← unchanged: retrospective output └── sessions/ ← unchanged: JSONL logs ``` This layout is a valid Obsidian vault. Open `brain/` as a vault and you get Wing-level folders in the file explorer, Hall subfolders, and Obsidian's graph view will show natural clusters by Wing. ### Obsidian integration properties - Every note uses YAML frontmatter with `wing`, `hall`, `created_at`, and optional `tags` - Wikilinks (`[[other-note]]`) written by `retrospective` or `brain_write` will resolve natively in Obsidian - `_index.md` files at Wing level act as MOC (Map of Content) — auto-generated, listing all notes in that Wing with their Hall - No Obsidian plugins required — the layout works with vanilla Obsidian --- ## Implementation ### 1. New path resolver: `ingestion/internal/brain/path.go` ```go package brain // ValidHalls is the closed vocabulary of hall names. var ValidHalls = map[string]bool{ "facts": true, "decisions": true, "failures": true, "hypotheses": true, "sources": true, } // NotePath resolves a canonical filesystem path for a note given wing, hall, // and slug. Returns error if hall is not in ValidHalls. func NotePath(brainDir, wing, hall, slug string) (string, error) ``` Wing and slug are sanitised (lowercase, alphanumeric + hyphens, no path separators). Hall is validated against `ValidHalls`. ### 2. Extend `WriteNote` in `ingestion/internal/api/handler.go` Add optional `wing` and `hall` fields to `writeRequest`: ```go type writeRequest struct { Content string `json:"content"` Filename string `json:"filename,omitempty"` Type string `json:"type,omitempty"` // kept for backwards compat Domain string `json:"domain,omitempty"` // kept for backwards compat Wing string `json:"wing,omitempty"` Hall string `json:"hall,omitempty"` } ``` When `wing` + `hall` are provided, route to `brain/wiki/<wing>/<hall>/<filename>` via `brain.NotePath`. When absent, fall back to current `brain/knowledge/` behaviour. This is fully backwards-compatible — no existing callers break. Inject YAML frontmatter automatically: ```yaml --- wing: jepa-fx hall: decisions created_at: 2026-05-07T10:00:00Z --- ``` ### 3. Extend `search.Query` in `ingestion/internal/search/search.go` Add `hall` and `wing` filter parameters: ```go type QueryOptions struct { Query string Limit int Wing string // optional: restrict to brain/wiki/<wing>/ Hall string // optional: restrict to brain/wiki/<wing>/<hall>/ } func Query(brainDir string, opts QueryOptions) ([]Result, error) ``` Path-based filtering: when `wing` is set, only walk `brain/wiki/<wing>/`. When `hall` is additionally set, only walk `brain/wiki/<wing>/<hall>/`. This is O(1) directory scoping — no change to the scoring logic. Update `Result` to include `wing` and `hall` fields extracted from frontmatter or path segments. ### 4. Update MCP tool schemas In `ingestion/internal/mcp/`: `brain_write` — add optional `wing` (string) and `hall` (enum: facts/decisions/failures/hypotheses/sources) parameters. `brain_query` — add optional `wing` (string) and `hall` (enum) parameters. ### 5. Index generation: `brain_index` New MCP tool (and REST endpoint `POST /index`) that walks `brain/wiki/` and regenerates `_index.md` at each Wing directory: ```markdown # jepa-fx | Hall | Note | Created | |------|------|---------| | decisions | [val-vol-r2-as-optimization-metric](decisions/val-vol-r2-as-optimization-metric.md) | 2026-05-06 | | facts | [lejpa-architecture](facts/lejpa-architecture.md) | 2026-05-04 | ``` Called automatically after every `brain_write` with `wing`+`hall`, and available as a manual trigger. ### 6. Migration script: `scripts/migrate-brain-halls.sh` One-time script to migrate existing `brain/wiki/concepts/` and `brain/wiki/entities/` files into the new layout. Reads existing frontmatter `type:` and `domain:` fields to infer Wing and Hall. Falls back to `wing=general`, `hall=facts` for unmapped files. Non-destructive: leaves originals in place until `--commit` flag is passed. --- ## Acceptance criteria - `POST /write` with `{"wing":"jepa-fx","hall":"decisions","content":"...","filename":"my-decision"}` writes to `brain/wiki/jepa-fx/decisions/my-decision.md` with correct frontmatter - `POST /write` without `wing`/`hall` continues to write to `brain/knowledge/` unchanged - `POST /query` with `{"query":"lejpa","wing":"jepa-fx","hall":"facts"}` only searches `brain/wiki/jepa-fx/facts/` - `brain_write` MCP tool accepts and passes through `wing` and `hall` - `brain_query` MCP tool accepts and passes through `wing` and `hall` - Opening `brain/` as an Obsidian vault shows Wing folders in the file explorer with Hall subfolders - `_index.md` at each Wing directory lists all notes in that Wing with Hall column - Migration script runs without error on an empty brain and on the existing brain fixture - All existing tests pass (backwards-compatible path) - New tests cover: valid wing+hall write, invalid hall rejected, wing-scoped query returns only wing results ## Branch `feat/brain-halls` from `main` ## Out of scope - Vector/embedding search (separate issue) - Tunnel links between Wings (follow-up after Halls are stable) - Obsidian plugin development - Changing `brain/sessions/` or `brain/raw/` layout --- _Created via git-mcp on behalf of @mathiasbq_
mathias commented 2026-05-18 18:47:29 +00:00
Owner
Copy Link

Shipped in 75685e7. All acceptance criteria met.

What landed

New package ingestion/internal/brain/

  • ValidHalls, IsValidHall, Sanitise(s), NotePath(brainDir, wing, hall, slug) — path.go
  • BuildWingIndex(brainDir, wing), BuildAllWingIndexes(brainDir) — index.go
  • Full test coverage (path_test.go, index_test.go)

API surface

  • writeRequest gains wing, hall. WriteNote refactored to WriteNoteOptions. With wing+hall set → routes to brain/wiki/<wing>/<hall>/<slug>.md with wing/hall/created_at YAML frontmatter injected. Without → legacy brain/knowledge/ path preserved (no breaking change).
  • queryRequest gains wing, hall. search.Query now takes QueryOptions{Query, Limit, Wing, Hall}. Walks brain/wiki/<wing>/<hall>/ when scoped, both knowledge/ and wiki/ otherwise. Result carries wing/hall from frontmatter or path.
  • New POST /index REST endpoint.

MCP

  • brain_write schema: optional wing + hall (enum-validated).
  • brain_query schema: optional wing + hall (enum-validated).
  • New tool brain_index (optional wing arg; absent = rebuild all).
  • Auto: brain_write with wing+hall triggers BuildWingIndex on the affected wing.

Migration

  • scripts/migrate-brain-halls.sh — dry-run by default, --commit applies. Maps type: → hall (decision/hypothesis/failure/source → eponymous; concept/entity/other → facts), domain: → wing (sanitised; default general). Idempotent. Tested against synthetic brain: 3 notes migrate cleanly, re-run no-ops, empty brain handled.

Acceptance criteria

  • POST /write {wing,hall,content,filename} → brain/wiki/<wing>/<hall>/<file>.md with correct frontmatter
  • POST /write without wing/hall → brain/knowledge/ (legacy preserved)
  • POST /query {query,wing,hall} scopes search
  • brain_write MCP tool accepts wing/hall
  • brain_query MCP tool accepts wing/hall
  • Obsidian vault structure (file explorer shows wing folders, hall subfolders)
  • _index.md at each wing root listing notes with Hall|Note|Created columns
  • Migration script: empty brain ok, existing flat layout ok
  • All existing tests pass + new tests for valid wing+hall write, invalid hall rejected, wing-scoped query
  • task check clean

Out of scope

Tunnels → #16. Embeddings → #8. Both depend on this issue and are now unblocked.

Closing.

Shipped in `75685e7`. All acceptance criteria met. ### What landed **New package `ingestion/internal/brain/`** - `ValidHalls`, `IsValidHall`, `Sanitise(s)`, `NotePath(brainDir, wing, hall, slug)` — path.go - `BuildWingIndex(brainDir, wing)`, `BuildAllWingIndexes(brainDir)` — index.go - Full test coverage (path_test.go, index_test.go) **API surface** - `writeRequest` gains `wing`, `hall`. `WriteNote` refactored to `WriteNoteOptions`. With wing+hall set → routes to `brain/wiki/<wing>/<hall>/<slug>.md` with `wing/hall/created_at` YAML frontmatter injected. Without → legacy `brain/knowledge/` path preserved (no breaking change). - `queryRequest` gains `wing`, `hall`. `search.Query` now takes `QueryOptions{Query, Limit, Wing, Hall}`. Walks `brain/wiki/<wing>/<hall>/` when scoped, both `knowledge/` and `wiki/` otherwise. `Result` carries wing/hall from frontmatter or path. - New `POST /index` REST endpoint. **MCP** - `brain_write` schema: optional `wing` + `hall` (enum-validated). - `brain_query` schema: optional `wing` + `hall` (enum-validated). - New tool `brain_index` (optional `wing` arg; absent = rebuild all). - Auto: `brain_write` with wing+hall triggers `BuildWingIndex` on the affected wing. **Migration** - `scripts/migrate-brain-halls.sh` — dry-run by default, `--commit` applies. Maps `type:` → hall (decision/hypothesis/failure/source → eponymous; concept/entity/other → facts), `domain:` → wing (sanitised; default `general`). Idempotent. Tested against synthetic brain: 3 notes migrate cleanly, re-run no-ops, empty brain handled. ### Acceptance criteria - [x] `POST /write {wing,hall,content,filename}` → `brain/wiki/<wing>/<hall>/<file>.md` with correct frontmatter - [x] `POST /write` without wing/hall → `brain/knowledge/` (legacy preserved) - [x] `POST /query {query,wing,hall}` scopes search - [x] `brain_write` MCP tool accepts wing/hall - [x] `brain_query` MCP tool accepts wing/hall - [x] Obsidian vault structure (file explorer shows wing folders, hall subfolders) - [x] `_index.md` at each wing root listing notes with Hall|Note|Created columns - [x] Migration script: empty brain ok, existing flat layout ok - [x] All existing tests pass + new tests for valid wing+hall write, invalid hall rejected, wing-scoped query - [x] `task check` clean ### Out of scope Tunnels → #16. Embeddings → #8. Both depend on this issue and are now unblocked. Closing.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
mathias
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mathias/hyperguild#1
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?