Single-Writer SQLite/Yjs Architecture

NOXR-eueslset — branch: single-write-hail-mary

This branch eliminated the dual-write path (markdown + SQLite) from fp-core. SQLite/Yjs is now the single source of truth. The desktop app reads from the database via main-process IPC, detects cross-process changes via PRAGMA data_version polling, and uses optimistic mutations for instant UI feedback. Y.Doc compaction keeps the update table bounded.

1 What Changed
Removed
  • Markdown file writes for issues and comments
  • fs.watch() file watcher in desktop app
  • Post-sync materialization to markdown
  • experimental_issue_db feature flag
  • Legacy fp db command group
Added
  • DatabasePoller (PRAGMA data_version)
  • Optimistic mutation helper with seq-guarded rollback
  • Y.Doc snapshot compaction (threshold: 50 updates)
  • Denormalized issues_index table
  • Granular ISSUES_PATCH events
Key Decisions
  • No Y.Docs in renderer — main process only
  • Renderer sends edit intents over IPC
  • Zustand remains reactive store in renderer
  • 300ms poll interval for cross-process detection
  • Monotonic seq for gap detection
2 System Topology
flowchart TD
  subgraph CLI["CLI Process (apps/fp)"]
    CLIW["writes.ts\ncreateIssue / updateIssue / deleteIssue"]
    CLISYNC["syncIssues\none-shot sync"]
  end

  subgraph Desktop["Desktop App (apps/desktop)"]
    subgraph Main["Main Process"]
      IPC["IPC Handler\nissue-handler.ts"]
      OPT["optimisticMutation\nseq-guarded broadcast"]
      WATCHER["DatabasePoller\nPRAGMA data_version\n300ms poll"]
      PUSH["enqueuePushUpdates\nbuffered push queue"]
    end

    subgraph Sync["Sync Utility Process"]
      SLOOP["Sync Loop\n5min dormant cycle"]
      SOPS["syncIssues\nCRDT merge"]
    end

    subgraph Renderer["Renderer Process"]
      ZUST["Zustand Store\nissues map + details"]
      HOOK["useIssueSync\nevent listener"]
    end
  end

  subgraph Storage["SQLite — fp.db"]
    YJS["yjs_updates\nbinary CRDT deltas"]
    SNAP["yjs_snapshots\ncompacted state"]
    IDX["issues_index\ndenormalized cache"]
  end

  SERVER["Fiberplane Server\nFpProjectSyncDO"]

  CLIW -->|"saveUpdateStrict + upsertIndex"| YJS
  CLIW -->|"best-effort"| IDX
  CLISYNC <-->|"WebSocket handshake"| SERVER

  IPC -->|"edit intent"| OPT
  OPT -->|"ISSUES_PATCH (optimistic)"| HOOK
  OPT -->|"persist via fp-core"| YJS
  OPT -->|"issue IDs"| PUSH
  PUSH -->|"PUSH_UPDATES msg"| SLOOP

  SLOOP --> SOPS
  SOPS <-->|"WebSocket sync"| SERVER
  SOPS -->|"saveLocalUpdate"| YJS
  SOPS -->|"SYNC_UPDATE / NEW_ISSUE"| IPC

  WATCHER -->|"poll data_version"| YJS
  WATCHER -->|"ISSUES_PATCH (cross-proc)"| HOOK

  HOOK --> ZUST

  classDef cli fill:#b5761422,stroke:#b57614,stroke-width:2px
  classDef main fill:#1e3a5f22,stroke:#1e3a5f,stroke-width:2px
  classDef sync fill:#6e3a6e22,stroke:#6e3a6e,stroke-width:2px
  classDef renderer fill:#0e7c6b22,stroke:#0e7c6b,stroke-width:2px
  classDef storage fill:#4a7c3f22,stroke:#4a7c3f,stroke-width:2px
  classDef server fill:#9e3a1a22,stroke:#9e3a1a,stroke-width:2px

  class CLIW,CLISYNC cli
  class IPC,OPT,WATCHER,PUSH main
  class SLOOP,SOPS sync
  class ZUST,HOOK renderer
  class YJS,SNAP,IDX storage
  class SERVER server
    
CLI
Main Process
Sync Worker
Renderer
SQLite
Server
3 Write Paths
CLI Write Path
  1. 1User runs fp issue create or fp issue update
  2. 2Calls createIssue() / updateIssue() in writes.ts
  3. 3Load Y.Doc from SQLite, mutate in memory
  4. 4Compute delta: Y.encodeStateAsUpdate(doc, sv)
  5. 5saveUpdateStrict() — INSERT into yjs_updates
  6. 6upsertIndex() — UPDATE issues_index (best-effort)
  7. 7Recompute child counts if parent changed
Cross-process effect: CLI writes increment SQLite's PRAGMA data_version, which the desktop's DatabasePoller detects within ~350ms (300ms poll + 50ms settle).
Desktop Write Path
  1. 1Renderer sends edit intent via IPC RPC
  2. 2Main process handler calls optimisticMutation()
  3. 3Build optimistic IssueSummary, bump patchSeq
  4. 4Broadcast ISSUES_PATCH immediately — <1ms
  5. 5Persist to SQLite in background (same writes.ts path)
  6. 6Call skipNextUpdate(issueId) on DatabasePoller
  7. 7Enqueue PUSH_UPDATES to sync worker
Rollback guard: If persist fails and patchSeq hasn't advanced (no concurrent edits), the optimistic patch is rolled back to cached state.
4 Database Watching

How cross-process changes reach the UI

sequenceDiagram
  participant CLI as CLI Process
  participant DB as fp.db (SQLite)
  participant POLL as DatabasePoller
  participant MAIN as Main Process
  participant REND as Renderer

  CLI->>DB: saveUpdateStrict INSERT yjs_updates
  Note over DB: data_version increments

  loop Every 300ms
    POLL->>DB: PRAGMA data_version
    POLL->>POLL: Compare with last known version
  end

  POLL->>DB: Detected change! Wait 50ms settle
  POLL->>DB: SELECT id, issue_id FROM yjs_updates WHERE id > lastSeenId
  POLL->>POLL: Filter out own-process writes via recentOwnWrites map

  POLL->>MAIN: Emit issues-patched event with changedIds
  MAIN->>MAIN: Read full issues from SQLite
  MAIN->>MAIN: Build IssueSummary array, bump patchSeq
  MAIN->>REND: Broadcast ISSUES_PATCH with seq + updates
  REND->>REND: useIssueSync merges into Zustand store
  REND->>REND: React re-renders with new data
    
DatabasePoller Details
ParameterValue
Poll interval300ms
Settle delay50ms
Own-write suppress window750ms
Max consecutive errors10 (then stops)
Ref countingMultiple windows can watch same DB
Own-write suppression

When the main process writes to SQLite (via optimisticMutation), it calls skipNextUpdate(issueId) which adds the issue ID to a recentOwnWrites map with a 750ms TTL. The poller filters these out during processChanges() to prevent re-broadcasting changes the main process already broadcast optimistically.

Why PRAGMA data_version? — SQLite's sqlite3_update_hook only fires for same-connection writes. Since the CLI runs in a separate process with its own connection, PRAGMA data_version (which changes whenever any connection modifies the database) is the only reliable cross-process change detector. It's extremely cheap — no table scan, just reading a counter from the database header.
5 Remote Sync Flow
flowchart TD
  subgraph Push["Push: Local to Server"]
    A1["Local write completes"] --> A2["enqueuePushUpdates\nbuffer issue IDs"]
    A2 --> A3["PUSH_UPDATES message\nto sync worker"]
    A3 --> A4["syncIssues one-shot\nWebSocket handshake"]
    A4 --> A5["Server merges CRDT"]
  end

  subgraph Pull["Pull: Server to Local"]
    B1["Sync loop triggers\n5min dormant cycle"] --> B2["syncIssues one-shot\nload local SVs"]
    B2 --> B3["Server sends new updates"]
    B3 --> B4["saveLocalUpdate\nINSERT yjs_updates + reindex"]
    B4 --> B5["SYNC_UPDATE message\nto main process"]
    B5 --> B6["handleSyncIssueUpdate\nskipNextUpdate + broadcast"]
    B6 --> B7["ISSUES_PATCH\nto all renderers"]
  end

  classDef push fill:#4a7c3f22,stroke:#4a7c3f,stroke-width:2px
  classDef pull fill:#6e3a6e22,stroke:#6e3a6e,stroke-width:2px

  class A1,A2,A3,A4,A5 push
  class B1,B2,B3,B4,B5,B6,B7 pull
    
Sync Callbacks (fp-core)
The sync worker provides 4 callbacks to syncIssues(): loadLocalDoc loads Y.Doc from SQLite, saveLocalUpdate saves delta + reindexes, createLocalDoc creates new Y.Doc, persistSyncHash stores state vector hash for next handshake.
Avoiding Double Processing
When sync saves remote updates to SQLite, it sends SYNC_UPDATE to main. Main calls skipNextUpdate() before broadcasting, so the DatabasePoller won't re-process those same rows. Without this, every synced change would be broadcast twice.
6 Y.Doc Compaction
Compaction Pipeline
Capture max ID
MAX(id) from
yjs_updates
→
Count updates
Skip if fewer
than threshold
→
Load full Y.Doc
Apply snapshot
+ all updates
→
Encode snapshot
Y.encodeStateAsUpdate
single blob
→
Transaction
UPSERT snapshot
DELETE old updates
When does it trigger?
TriggerDetails
Desktop boot getNodeDbLayer() launches compactAllSnapshots({ threshold: 50 }) as a fire-and-forget daemon
Auto-rerun Per-storageDir cooldown: 10 min after success, 2 min after failure
CLI manual fp project db compact command triggers on demand
Max retries 2 retries with exponential backoff per storageDir
Concurrency safety

Compaction captures MAX(id) before reading, so concurrent writes that arrive during compaction are preserved. The final DELETE only removes updates WHERE id <= maxUpdateId.

The snapshot stores last_update_id as a foreign key. loadIssue() reads the snapshot and then applies only updates WHERE id > last_update_id, so it always sees a consistent state.

An idempotency guard skips if another process already compacted past this point.

7 IPC Event Architecture
Event Flow
Event Source Payload Renderer Action
ISSUES_PATCH optimisticMutation, DatabasePoller, sync handler { seq, updates: IssueSummary[], deleted: string[] } Merge into Zustand issues map, check seq for gaps
ISSUES_CHANGED Bulk fallback (sync complete with newForClient > 0) undefined Full refetch of all issues
COMMENTS_UPDATED Comment create/delete handlers { issueId } Invalidate detail cache, refetch comments
ISSUE_DETAIL_UPDATED Story mutations, detail changes { issueId } Refetch issue detail
SYNC_STATUS_CHANGED Sync status store { projectStatus } Update SyncStatusIndicator UI
Sequence Gap Detection

The main process maintains a monotonically increasing patchSeq counter in AppState. Every ISSUES_PATCH event carries the current seq.

The renderer's useIssueSync hook compares each incoming seq against its last known value. If it detects a gap (missed patches), it schedules a full sync on next window focus to recover.

// useIssueSync.ts — gap detection const onSeqUpdate = (seq: number) => { const expected = lastSeq + 1; if (seq > expected) { // Missed patches — schedule full sync scheduleFullSync(); } setLastSeq(seq); };
8 Key Files
Component File Key exports
Core write logic packages/fp-core/src/storage/writes.ts createIssue, updateIssue, deleteIssue, createComment
Y.Doc storage packages/fp-core/src/storage/database/yjs-docs.ts saveUpdateStrict, loadIssue, compactIssueSnapshot, compactAllSnapshots
Issues index packages/fp-core/src/storage/database/issues-index.ts upsertIndex, listFromIndex, reindexAll, detectStaleIndex
DatabasePoller apps/desktop/src/main/watcher.ts DatabasePoller, watch, poll, processChanges, skipNextUpdate
IPC handlers apps/desktop/src/main/rpc/issue-handler.ts handleIssueCreate, handleIssueUpdate, handleCommentAdd
Optimistic helper apps/desktop/src/main/rpc/optimistic-mutation.ts optimisticMutation
Push queue apps/desktop/src/main/push-updates.ts enqueuePushUpdates, flushPendingPushUpdates
Sync operation apps/desktop/src/main/sync/sync-operation.ts ProjectSyncOperationLive
IPC event types apps/desktop/src/shared/ipc.ts IPC_EVENTS, IpcEventPayloadMap, IpcHandlerMap
Renderer sync hook apps/desktop/src/renderer/hooks/useIssueSync.ts useIssueSync
Zustand issue store apps/desktop/src/renderer/store/issues.ts setIssues, mergeIssues, updateDetail
Desktop DB init apps/desktop/src/main/database.ts getNodeDbLayer (launches compaction daemon)