mnemosyne/ROADMAP.md

Implementation Roadmap

Overview

9 phases, incrementally building from a working Pichay fork to the full object-addressed memory system. Each phase produces a testable, usable artifact.

Estimated total effort: 8-12 weeks for a solo developer.

---

Phase 1: Pichay Baseline (Week 1)

Goal: Get the existing Pichay proxy running between opencode and Anthropic API. Validate structural waste reduction on real sessions.

Tasks

• 1.1 Fork fsgeek/pichay at tag v0.1.0-paper (commit b56701a)
• 1.2 Set up development environment
  • Python 3.11+, asyncio, httpx
  • Local opencode instance
  • Proxy configuration: ANTHROPIC_BASE_URL=http://localhost:8080
• 1.3 Run proxy in passthrough mode (no eviction)
  • Verify: opencode works normally through proxy
  • Log: request/response sizes, token counts, tool call inventory
• 1.4 Enable FIFO eviction (Pichay default settings)
  • tau = 4 turns, s_min = 500 bytes
  • Verify: tombstones appear for old tool results
  • Measure: tokens saved, fault rate
• 1.5 Record 5+ real coding sessions through the proxy
  • Use probe.py to generate session analytics
  • Baseline metrics: waste %, amplification factor, fault rate
• 1.6 Write session replay infrastructure
  • Record full message traces (request + response pairs)
  • Replay tool for offline testing of later phases

Deliverable

Working proxy with FIFO eviction. Baseline metrics on real sessions.

Success Criteria

• Proxy is transparent (opencode works identically)
• Measurable token reduction (target: >15%)
• Fault rate < 0.1%
• 5+ recorded sessions for replay testing

---

Phase 2: Multi-Fidelity Eviction (Weeks 2-3)

Goal: Replace binary eviction (resident/tombstone) with graduated fidelity levels. Introduce the Helper LLM for summary generation.

Tasks

• 2.1 Implement the Fidelity Manager state machine
  • States: L0 (full), L1 (detailed summary), L2 (compact summary), L3 (stub), L4 (evicted)
  • Transitions: degrade (pressure), upgrade (access/fault), pin (fault-driven)
  • In-memory state per session (no DB yet)
• 2.2 Implement pressure zones
  • Normal (<50%), Caution (50-70%), Warning (70-85%), Critical (85-95%), Emergency (>95%)
  • Token counting per fidelity level
  • Configurable thresholds via TOML config
• 2.3 Integrate Helper LLM (Anthropic Haiku)
  • API client with retry, timeout, error handling
  • Prompt template for L0 -> L1 summarization (detailed summary + declared losses)
  • Prompt template for L1 -> L2 compression
  • Response parsing: extract summary, losses, can_answer, key_entities
• 2.4 Implement fidelity degradation on pressure
  • Walk objects oldest-first
  • Generate summaries via Helper LLM on transition
  • Replace content in message array with summary + loss declaration
  • Format: [Summary of {type}: {stub}]\n{summary}\n[Cannot answer: {losses}]
• 2.5 Implement fidelity upgrade on access
  • When model references an L1/L2/L3 object (detected by content overlap or tool call), upgrade to L0
  • Restore full content from in-memory cache (client history is the backing store)
• 2.6 Extend fault detection for fidelity-aware faults
  • L3 stub referenced -> upgrade to L1 (not necessarily L0)
  • L2 compact referenced -> upgrade to L1
  • Only full page-in if model explicitly re-requests (tool call match)
• 2.7 Add declared losses to eviction tombstones
  • Format: [Paged out: {stub}. Lost: {losses}. Restore if you need: {fault_when}]
• 2.8 Replay testing against Phase 1 baseline
  • Compare: token reduction, fault rate, summary quality
  • Manual review: are declared losses accurate?

Deliverable

Multi-fidelity proxy with Helper LLM summarization and declared losses.

Success Criteria

• Token reduction > 40% (up from Phase 1's ~20%)
• Fault rate < 0.05% (better than binary eviction -- summaries prevent unnecessary faults)
• Helper LLM cost < $0.01 per session
• Declared losses are accurate in >90% of spot checks

---

Phase 3: Queryable Backing Store + Micro-Faults (Weeks 4-5)

Goal: Add PostgreSQL + pgvector as persistent backing store. Implement memory_query phantom tool for micro-faults. This is the single highest-impact feature.

Tasks

• 3.1 Set up PostgreSQL + pgvector
  • Docker Compose for local dev
  • Schema from SCHEMA.md (sessions, semantic_objects tables)
  • Connection pooling (asyncpg)
• 3.2 Implement Object Store module
  • CRUD operations for semantic_objects
  • Embedding generation (all-MiniLM-L6-v2 via sentence-transformers, ONNX runtime)
  • Store objects on creation (every tool result, message span)
  • Full content always preserved in DB regardless of context fidelity
• 3.3 Implement semantic search
  • Vector similarity search (pgvector cosine distance)
  • Full-text search (PostgreSQL tsvector)
  • Hybrid search (weighted combination)
  • Metadata-filtered search (by type, tags, date range)
• 3.4 Implement memory_query phantom tool
  • Add tool definition to phantom tool injection
  • Intercept from streaming response
  • Flow: intercept -> query Object Store -> send top-k results to Helper LLM -> inject answer
  • Synthetic tool result format: [Memory Query Result]\nQ: {question}\nA: {answer}\n[Source: {object stubs}]
• 3.5 Implement memory_restore phantom tool
  • Full page-in from backing store (traditional fault)
  • Upgrade object to L0
  • Trigger pressure-based eviction of other objects if needed
• 3.6 Implement memory_release phantom tool
  • Model voluntarily releases objects
  • Immediate degradation to L3 or L4
  • Log cooperative eviction event
• 3.7 Implement phantom tool injection
  • Add phantom tool definitions to the model's tool list in each request
  • Parse phantom tool calls from streaming response BEFORE client receives them
  • Handle phantom tool results transparently
• 3.8 Measure micro-fault effectiveness
  • Track: micro-faults attempted, successful (model didn't need to restore after), tokens saved per micro-fault
  • Compare: micro-fault answer quality vs full restore (LLM-judged)

Deliverable

Full proxy with persistent backing store, semantic search, and micro-fault capability.

Success Criteria

• Token reduction > 60%
• Micro-fault success rate > 85% (model doesn't need full restore after micro-fault)
• Micro-fault latency < 500ms (embedding + DB query + Helper LLM)
• Token savings per micro-fault: >95% vs full restore
• DB query latency < 50ms for semantic search

---

Phase 4a: Object Segmentation (Week 6)

Goal: Replace file-path-based page identity with semantic object detection. Conversations are segmented into coherent objects with types and relationships.

Tasks

• 4a.1 Implement structural segmentation (Strategy A)
  • Rule-based: tool call boundaries -> tool_result / file_context objects
  • User turn + assistant response = conversation span
  • Topic shift detection: embedding cosine between consecutive spans < 0.7 -> new object
  • Object type classification: rules based on tool name, content patterns
    • Read tool -> file_context
    • Bash/Grep tool -> tool_result
    • Error in output -> error_context
    • "I'll implement..." / plan language -> plan
    • "Let's use X because Y" / decision language -> design_decision
• 4a.2 Implement object type classifier
  • Helper LLM or simple keyword-based classifier
  • Input: content span + preceding context
  • Output: object_type + stub + tags
  • Latency budget: <100ms (prefer rules, fallback to Helper LLM)
• 4a.3 Retroactive segmentation
  • On session start: no segmentation (objects created per-tool-result)
  • Every 10 turns: re-examine recent objects, merge small ones, split large ones
  • Merge criterion: consecutive objects of same type with embedding similarity > 0.8
  • Split criterion: single object > 5000 tokens with internal topic shift
• 4a.4 Object deduplication
  • file_context: dedup by source_key (file path). New read supersedes old.
  • tool_result: no dedup (each is unique)
  • conversation_phase: no dedup
  • Add 'supersedes' relationship when replacing
• 4a.5 Store relationships
  • Automatic: file_context referenced in debugging_session -> 'references' edge
  • Automatic: design_decision made during conversation_phase -> 'parent_of' edge
  • Detection: key_entities overlap between objects suggests relationship
• 4a.6 Test segmentation quality
  • Replay recorded sessions
  • Manual review: do objects correspond to intuitive "chunks" of work?
  • Measure: average object size, type distribution, relationship density

Deliverable

Proxy segments conversations into typed semantic objects with relationships.

Success Criteria

• Objects correspond to intuitive conversation segments (manual review)
• Average object size: 500-3000 tokens (not too fine, not too coarse)
• Type classification accuracy > 85%
• Segmentation latency < 50ms per turn (structural) or < 200ms (semantic)

---

Phase 4b: Object Relationships + Co-Fidelity (Week 7)

Goal: Use relationships between objects to make smarter fidelity decisions. When one object is upgraded, related objects are considered for upgrade too.

Tasks

• 4b.1 Implement relationship-aware fidelity management
  • When upgrading object X to L0, check depends_on and references edges
  • If related object Y is at L2+, upgrade to L1 (not L0 -- don't over-promote)
  • Configurable: max relationship hops (default: 2), max co-upgrades (default: 3)
• 4b.2 Implement relationship-aware eviction
  • When degrading object X, DON'T degrade objects that X depends_on if they're actively referenced by other L0 objects
  • Eviction priority: objects with no inbound references or depends_on edges are evicted first (they're "leaf" objects)
• 4b.3 Relationship visualization (debug tool)
  • CLI command: mnemosyne graph --session <id>
  • Output: DOT graph of objects + relationships + fidelity levels
  • For debugging: identify orphaned objects, over-connected clusters
• 4b.4 Test co-fidelity management
  • Scenario: model starts working on auth -> auth objects at L0, related files at L1
  • Model switches to tests -> auth demoted to L1/L2, test objects promoted
  • Model returns to auth -> auth restored, relationships pull in dependencies

Deliverable

Relationship-aware fidelity management. Objects that belong together stay together.

Success Criteria

• Co-fidelity reduces fault rate by >20% vs independent fidelity management
• No "orphaned dependency" faults (model needs X, but X's dependency Y is evicted)

---

Phase 4c: Goal-Aware Retrieval (Week 8)

Goal: When the user's goal changes, proactively swap context. The Helper LLM reads the new goal, queries the Object Store, and assembles a focused context window.

Tasks

• 4c.1 Implement goal transition detection
  • Embed each user message
  • Compare to previous user message embedding (cosine similarity)
  • Threshold: < 0.5 = major topic shift, trigger goal-aware retrieval
  • Also detect explicit signals: "now let's work on...", "moving to...", "switching to..."
• 4c.2 Implement goal classification
  • Helper LLM call (~100ms): Input: current user message + last 2 turns of context Output: { goal, relevant_types, relevant_tags, predicted_needs }
  • Cache goal classification (don't re-classify if message is a follow-up)
• 4c.3 Implement context swap on goal change
  • Query Object Store: find top-20 objects by similarity to new goal
  • Rank by: embedding similarity * recency_weight * type_match_bonus
  • Assemble new context:
    • Always: system prompt, last 2 user turns, pinned objects
    • From goal query: top objects at appropriate fidelity (budget permitting)
    • Previously active but now irrelevant: degrade to L2/L3 (not L4 -- recent work)
• 4c.4 Implement predictive loading
  • Helper LLM predicts what the model will need for this goal
  • Pre-load predicted objects at L1 (ready for quick upgrade)
  • Track prediction accuracy: did the model actually access predicted objects?
• 4c.5 Test goal-aware retrieval
  • Scenario: multi-task session (auth -> tests -> docs -> bugfix)
  • Measure: context relevance at each goal transition
  • Compare: goal-aware vs naive (no swap) in token efficiency and fault rate

Deliverable

Proxy proactively loads relevant context on goal transitions.

Success Criteria

• Goal detection accuracy > 80% (catches real transitions, few false positives)
• Context relevance after swap > 70% (measured by: did the model use the loaded objects?)
• Predictive loading accuracy > 50% (better than random)
• No regressions in token efficiency or fault rate

---

Phase 4d: Admission Control (Week 9)

Goal: Not everything deserves to be a stored object. Score incoming content and reject low-value items to keep the Object Store clean.

Tasks

• 4d.1 Implement admission scorer
  • Four factors: TypePrior, Novelty, Utility, Recency
  • TypePrior: static weights per object_type (design_decision=1.0, tool_result=0.4, etc.)
  • Novelty: cosine distance to nearest existing object in session (>0.15 = novel)
  • Utility: Helper LLM scores future relevance 0-1 (or local model via Ollama)
  • Recency: exponential decay from turn number
  • Configurable weights (default from A-MAC: T=0.35, N=0.25, U=0.25, R=0.15)
• 4d.2 Implement admission threshold
  • Default: 0.4
  • Items below threshold: not stored in Object Store
  • Still present in client's unmodified history (Pichay backing store)
  • Log rejected items for threshold tuning
• 4d.3 Implement rejection patterns
  • Always reject: ls output, git status, routine directory listings
  • Always reject: duplicate file reads where content hash matches existing object
  • Always reject: purely procedural assistant responses ("Sure, I'll do that")
  • Configurable: rejection rules in TOML config
• 4d.4 Threshold tuning
  • Replay recorded sessions with different thresholds
  • Find threshold that minimizes (false rejections * fault_cost + storage * store_cost)
  • Log admission_scores table for analysis

Deliverable

Admission gate filters low-value content from the Object Store.

Success Criteria

• 30% of tool results rejected (routine/duplicate content)

• No false rejections that cause faults later (measure: rejected items that model would have needed, detected via fault-after-reject tracking)
• Object Store grows linearly with session complexity, not session length

---

Phase 4e: Entropy-Gated Faulting (Week 10)

Goal: Use the model's token-level entropy during generation as an automatic signal to inject more context. Complements declared losses.

Tasks

• 4e.1 Implement logprob extraction from streaming response
  • Anthropic API: stream_options.include_logprobs (if available)
  • Parse token logprobs from SSE stream in real-time
  • Calculate rolling entropy: H = -sum(p * log(p)) over top-k logprobs
• 4e.2 Implement entropy monitoring
  • Rolling window: last 20 tokens
  • Thresholds: normal (H < 1.5), elevated (1.5-2.2), high (H > 2.2)
  • Debounce: don't trigger on single high-entropy token (require 3+ consecutive)
• 4e.3 Implement entropy-triggered context injection
  • On elevated entropy: check if any L2/L3 objects match current generation topic
    • Extract current generation context (last 50 tokens)
    • Embed and search Object Store
    • If match found: silently upgrade to L1 in NEXT request (can't modify current)
  • On high entropy: more aggressive -- prepare micro-fault answer for likely question
• 4e.4 Evaluate entropy signal reliability
  • Compare: entropy at points where model made errors vs correct generation
  • Calibrate thresholds per model (Opus vs Sonnet vs Haiku have different baselines)
  • Measure: false positive rate (elevated entropy but model was fine)

Deliverable

Proxy monitors generation entropy and proactively loads context when model is uncertain.

Success Criteria

• Entropy signal detects genuine uncertainty >70% of the time
• False positive rate < 30% (elevated entropy that didn't need intervention)
• Measurable quality improvement on tasks where entropy-gating activated
• Note: This is the most experimental phase. Success criteria may be revised.

Fallback

If logprobs are not reliably available from the API, this phase can be deferred. The system works well without it -- declared losses + manual faulting cover most cases.

---

Phase 5: OpenCode Plugin Integration (Week 11)

Goal: Package as an oh-my-opencode plugin with a companion MCP server for user-facing memory inspection tools.

Tasks

• 5.1 Create oh-my-opencode plugin package
  • npm package: opencode-mnemosyne
  • Hook: experimental.chat.messages.transform for context assembly
  • Hook: experimental.session.compacting for custom compaction
  • Hook: tool.execute.after for object creation on tool results
  • Configuration via oh-my-opencode.json
• 5.2 Create MCP server for user-facing tools
  • memory_stats: show current context pressure, object counts by type/fidelity
  • memory_objects: list all objects with fidelity, type, stub
  • memory_inspect <id>: show object detail (all fidelity levels, losses, relationships)
  • memory_graph: show object relationship graph
  • memory_config: view/update runtime configuration
• 5.3 Documentation
  • Installation guide
  • Configuration reference
  • Troubleshooting guide
  • Architecture overview for contributors
• 5.4 Session dashboard (optional)
  • Local web UI (served by proxy) showing:
    • Real-time context pressure gauge
    • Object timeline (creation, fidelity transitions, faults)
    • Token savings over time
    • Fault log

Deliverable

Installable plugin + MCP server. Users can inspect and configure memory behavior.

---

Phase 6: Semantic Segmentation + xMemory Hierarchy (Week 12)

Goal: Replace rule-based segmentation with xMemory's hierarchical approach for long sessions. Build the full messages -> episodes -> semantics -> themes hierarchy.

Tasks

• 6.1 Implement xMemory-style sparsity-semantics segmentation
  • Embed all messages in a session with all-MiniLM-L6-v2
  • Cluster by coherence using the sparsity-semantics objective: maximize inter-cluster diversity, minimize intra-cluster redundancy
  • Output: episodes (coherent sub-conversations)
• 6.2 Build hierarchy
  • Level 0: individual messages/tool results (existing objects)
  • Level 1: episodes (groups of related objects, from clustering)
  • Level 2: semantics (abstract themes spanning multiple episodes)
  • Level 3: themes (top-level categories for the entire session)
• 6.3 Hierarchical retrieval
  • Top-down: query matches theme -> expand to semantics -> expand to episodes -> expand to objects
  • Only expand when similarity score justifies it (reader uncertainty reduction)
  • Prevents redundant retrieval (a key xMemory advantage over flat search)
• 6.4 Incremental hierarchy maintenance
  • Don't rebuild from scratch every turn
  • New objects: assign to nearest episode, update episode embedding
  • Every 20 turns: re-cluster to catch topic drift
  • Major goal change: full re-hierarchy
• 6.5 Hierarchy-aware fidelity management
  • When an episode is at L2, all its objects are at L2 or lower
  • Upgrading an episode promotes its most relevant objects to L1
  • Themes can have their own summaries (super-summaries of episode summaries)

Deliverable

Full hierarchical segmentation for long sessions (>50 turns).

Success Criteria

• Retrieval quality improves for sessions >100 turns (measured by LLM-judged relevance)
• Hierarchy reduces redundancy in retrieved context (measured by token overlap between results)
• Incremental maintenance is fast (<500ms per turn)

---

Dependencies Between Phases

Phase 1 (Pichay baseline)
    |
Phase 2 (Multi-fidelity + Helper LLM)
    |
Phase 3 (Backing store + micro-faults)
   / \
  /   \
4a    4d (can run in parallel)
(segmentation)  (admission control)
  |
4b (relationships + co-fidelity)
  |
4c (goal-aware retrieval)
  |
4e (entropy-gated faulting) -- optional, experimental
  |
Phase 5 (plugin integration)
  |
Phase 6 (xMemory hierarchy)

Phases 4a-4e can be partially parallelized:

• 4a + 4d can be built simultaneously
• 4b depends on 4a
• 4c depends on 4a + 4b
• 4e is independent (depends only on Phase 3)
• Phase 5 can start after Phase 3 (plugin wrapping doesn't need 4a-4e)
• Phase 6 depends on 4a (needs basic segmentation first)

---

Risk Register

Risk	Likelihood	Impact	Mitigation
Anthropic API doesn't expose logprobs for streaming	Medium	Phase 4e blocked	Phase 4e is optional. System works without entropy gating.
Helper LLM summaries lose critical info	Medium	Quality degradation	Declared losses + micro-faults as safety net. Spot-check auditing.
Proxy adds too much latency	Low	User experience	Helper LLM calls are async (don't block response). Summarization happens post-response.
pgvector search too slow at scale	Low	Micro-fault latency	IVFFlat index. For extreme scale, switch to dedicated vector DB (Qdrant).
Object segmentation too noisy	Medium	Poor fidelity decisions	Conservative defaults (larger objects). Rule-based segmentation is robust.
Phantom tool parsing from streaming response is fragile	Medium	Proxy breaks	Extensive testing on recorded sessions. Fallback: don't parse, let tool call through.
Model doesn't use cooperative eviction (memory_release)	High	Reduced savings	Cooperative eviction is bonus. Pressure-based eviction works without model cooperation.
Cross-session memory introduces stale/wrong context	Medium	Wrong answers	Phase 6+ only. Confidence decay on persistent objects.