Single-Writer SQLite/Yjs Architecture
NOXR-eueslset — branch: single-write-hail-mary
This branch eliminated the dual-write path (markdown + SQLite) from fp-core. SQLite/Yjs is now the single source of truth. The desktop app reads from the database via main-process IPC, detects cross-process changes via PRAGMA data_version polling, and uses optimistic mutations for instant UI feedback. Y.Doc compaction keeps the update table bounded.
- Markdown file writes for issues and comments
- fs.watch() file watcher in desktop app
- Post-sync materialization to markdown
experimental_issue_dbfeature flag- Legacy
fp dbcommand group
- DatabasePoller (PRAGMA data_version)
- Optimistic mutation helper with seq-guarded rollback
- Y.Doc snapshot compaction (threshold: 50 updates)
- Denormalized
issues_indextable - Granular ISSUES_PATCH events
- No Y.Docs in renderer — main process only
- Renderer sends edit intents over IPC
- Zustand remains reactive store in renderer
- 300ms poll interval for cross-process detection
- Monotonic seq for gap detection
- 1User runs
fp issue createorfp issue update - 2Calls
createIssue()/updateIssue()in writes.ts - 3Load Y.Doc from SQLite, mutate in memory
- 4Compute delta:
Y.encodeStateAsUpdate(doc, sv) - 5
saveUpdateStrict()— INSERT intoyjs_updates - 6
upsertIndex()— UPDATEissues_index(best-effort) - 7Recompute child counts if parent changed
PRAGMA data_version,
which the desktop's DatabasePoller detects within ~350ms (300ms poll + 50ms settle).
- 1Renderer sends edit intent via IPC RPC
- 2Main process handler calls
optimisticMutation() - 3Build optimistic IssueSummary, bump
patchSeq - 4Broadcast
ISSUES_PATCHimmediately — <1ms - 5Persist to SQLite in background (same writes.ts path)
- 6Call
skipNextUpdate(issueId)on DatabasePoller - 7Enqueue
PUSH_UPDATESto sync worker
patchSeq hasn't advanced
(no concurrent edits), the optimistic patch is rolled back to cached state.
How cross-process changes reach the UI
| Parameter | Value |
|---|---|
| Poll interval | 300ms |
| Settle delay | 50ms |
| Own-write suppress window | 750ms |
| Max consecutive errors | 10 (then stops) |
| Ref counting | Multiple windows can watch same DB |
When the main process writes to SQLite (via optimisticMutation), it calls
skipNextUpdate(issueId) which adds the issue ID to a
recentOwnWrites map with a 750ms TTL. The poller filters these
out during processChanges() to prevent re-broadcasting changes
the main process already broadcast optimistically.
sqlite3_update_hook only fires
for same-connection writes. Since the CLI runs in a separate process with its own connection,
PRAGMA data_version (which changes whenever any connection modifies the database)
is the only reliable cross-process change detector. It's extremely cheap — no table scan, just
reading a counter from the database header.
syncIssues():
loadLocalDoc loads Y.Doc from SQLite,
saveLocalUpdate saves delta + reindexes,
createLocalDoc creates new Y.Doc,
persistSyncHash stores state vector hash for next handshake.
SYNC_UPDATE to main.
Main calls skipNextUpdate() before broadcasting, so the DatabasePoller
won't re-process those same rows. Without this, every synced change would be broadcast twice.
yjs_updates
than threshold
+ all updates
single blob
DELETE old updates
| Trigger | Details |
|---|---|
| Desktop boot | getNodeDbLayer() launches compactAllSnapshots({ threshold: 50 }) as a fire-and-forget daemon |
| Auto-rerun | Per-storageDir cooldown: 10 min after success, 2 min after failure |
| CLI manual | fp project db compact command triggers on demand |
| Max retries | 2 retries with exponential backoff per storageDir |
Compaction captures MAX(id) before reading, so concurrent writes that arrive
during compaction are preserved. The final DELETE only removes updates
WHERE id <= maxUpdateId.
The snapshot stores last_update_id as a foreign key.
loadIssue() reads the snapshot and then applies only updates
WHERE id > last_update_id, so it always sees a consistent state.
An idempotency guard skips if another process already compacted past this point.
| Event | Source | Payload | Renderer Action |
|---|---|---|---|
ISSUES_PATCH |
optimisticMutation, DatabasePoller, sync handler | { seq, updates: IssueSummary[], deleted: string[] } |
Merge into Zustand issues map, check seq for gaps |
ISSUES_CHANGED |
Bulk fallback (sync complete with newForClient > 0) | undefined |
Full refetch of all issues |
COMMENTS_UPDATED |
Comment create/delete handlers | { issueId } |
Invalidate detail cache, refetch comments |
ISSUE_DETAIL_UPDATED |
Story mutations, detail changes | { issueId } |
Refetch issue detail |
SYNC_STATUS_CHANGED |
Sync status store | { projectStatus } |
Update SyncStatusIndicator UI |
The main process maintains a monotonically increasing patchSeq counter
in AppState. Every ISSUES_PATCH event carries the current seq.
The renderer's useIssueSync hook compares each incoming seq against
its last known value. If it detects a gap (missed patches), it schedules a
full sync on next window focus to recover.
| Component | File | Key exports |
|---|---|---|
| Core write logic | packages/fp-core/src/storage/writes.ts |
createIssue, updateIssue, deleteIssue, createComment |
| Y.Doc storage | packages/fp-core/src/storage/database/yjs-docs.ts |
saveUpdateStrict, loadIssue, compactIssueSnapshot, compactAllSnapshots |
| Issues index | packages/fp-core/src/storage/database/issues-index.ts |
upsertIndex, listFromIndex, reindexAll, detectStaleIndex |
| DatabasePoller | apps/desktop/src/main/watcher.ts |
DatabasePoller, watch, poll, processChanges, skipNextUpdate |
| IPC handlers | apps/desktop/src/main/rpc/issue-handler.ts |
handleIssueCreate, handleIssueUpdate, handleCommentAdd |
| Optimistic helper | apps/desktop/src/main/rpc/optimistic-mutation.ts |
optimisticMutation |
| Push queue | apps/desktop/src/main/push-updates.ts |
enqueuePushUpdates, flushPendingPushUpdates |
| Sync operation | apps/desktop/src/main/sync/sync-operation.ts |
ProjectSyncOperationLive |
| IPC event types | apps/desktop/src/shared/ipc.ts |
IPC_EVENTS, IpcEventPayloadMap, IpcHandlerMap |
| Renderer sync hook | apps/desktop/src/renderer/hooks/useIssueSync.ts |
useIssueSync |
| Zustand issue store | apps/desktop/src/renderer/store/issues.ts |
setIssues, mergeIssues, updateDetail |
| Desktop DB init | apps/desktop/src/main/database.ts |
getNodeDbLayer (launches compaction daemon) |