paseo/docs/STORAGE_REVAMP_PLAN.md

Storage Revamp Plan

Status: active rollout, phases 1 and 2 complete

This document now tracks the storage revamp as it exists today, not as a speculative design exercise. The DB foundation and the project/workspace identity cutover have landed. What remains is the explicit creation/archive surface cleanup, timeline durability cutover, and final removal of legacy paths.

Goals

• make structured records durable in Drizzle + SQLite
• make projects and workspaces explicit first-class records
• stop deriving project/workspace identity from agent cwd
• keep agent snapshot persistence behind clear ownership
• move committed timeline history to storage-owned rows
• remove legacy JSON and in-memory authority once the DB path is proven

Out of scope

• moving config, keypairs, push tokens, or server identity into the DB
• persisting raw provider deltas or transport-only chunk streams
• designing a hosted/remote database story beyond keeping the schema portable
• durable reasoning history unless product explicitly asks for it later

Current state

The storage revamp is no longer hypothetical.

Completed:

• Drizzle + SQLite database bootstrap is in place
• projects, workspaces, and agent_snapshots use integer primary keys
• workspaces.project_id and agent_snapshots.workspace_id cascade on delete
• agent_snapshots.workspace_id is NOT NULL
• legacy JSON import feeds the DB-backed structured records
• project/workspace records use explicit directory fields instead of path-as-identity
• session read paths now use persisted workspace/project rows instead of cwd/git derivation
• workspace-reconciliation-service.ts is deleted
• workspace-registry-bootstrap.ts is deleted
• workspace-registry-model.ts is reduced to normalizeWorkspaceId

Still pending:

• explicit create_project / create_workspace API cleanup
• final archive cascade behavior for descendants and live agents
• committed timeline storage cutover
• removal of remaining legacy JSON and in-memory committed-history authority

Converged decisions

Structured record authority

Projects, workspaces, and agent snapshots are DB-backed structured records. The server should not recreate project/workspace identity from:

• git remotes
• worktree main-repo roots
• normalized cwd strings

Temporary exception:

• agent creation may still find-or-create a workspace by directory if the UI has not yet provided workspaceId explicitly

That fallback is transitional and should be deleted once the client always sends the workspace id.

Storage seams

The useful seams remain concrete and domain-shaped:

• ProjectRegistry
• WorkspaceRegistry
• AgentSnapshotStore
• AgentTimelineStore

There is no reason to reintroduce a reconciliation service layer for project/workspace identity.

Timeline contract

The long-term timeline contract remains:

• committed rows are durable, canonical history
• provisional live updates are transient subscription state
• committed history is fetched by seq
• provider history replay is not the durability mechanism

The structured-record cutover is complete before the timeline cutover so timeline rows can rely on stable DB-backed agent and workspace identity.

Remaining phases

Phase 3: Explicit creation and archive cleanup

Goal: Remove the last transitional write paths that still infer state from directories.

Required work:

• add explicit create_project handling
• add explicit create_workspace handling
• make agent creation require workspaceId once the UI is ready
• finish archive semantics for workspaces/projects and any descendant agent state
• remove the temporary find-or-create-by-directory fallback from agent creation

Exit gate:

• project/workspace creation is explicit end to end
• no normal creation path infers identity from cwd or git metadata
• archive flows behave consistently for structured records and live runtime state

Phase 4: Timeline storage cutover

Goal: Make committed history durable and storage-owned.

Required work:

• make AgentTimelineStore authoritative for committed history
• write one committed row per finalized logical item
• support tail, before-seq, and after-seq queries from storage
• stop treating provider history hydration as the normal refresh/load path
• keep provisional live updates in memory only

Exit gate:

• committed history survives daemon restart
• reconnect uses committed catch-up plus future live events without gaps or duplicates
• unloaded agents can serve committed history from storage alone

Phase 5: Legacy cleanup

Goal: Remove compatibility paths after the DB-backed model is fully authoritative.

Required work:

• remove legacy JSON authority for structured records
• remove in-memory committed-history ownership
• remove provider-history rehydrate compatibility paths
• trim dead protocol and reducer logic from the pre-storage model
• update architecture docs to match the final model

Exit gate:

• there is one durable storage path for structured records
• there is one durable storage path for committed timeline history
• the runtime no longer depends on the removed JSON/in-memory model

Data model summary

Projects

• integer primary key
• directory is unique
• display_name
• kind: git | directory
• optional git_remote
• timestamps and archive state

Workspaces

• integer primary key
• belongs to a project by project_id
• directory is unique
• display_name
• kind: checkout | worktree
• timestamps and archive state

Agent snapshots

• agent_id remains the primary key
• belongs to a workspace by integer workspace_id
• workspace_id is required
• timestamps, lifecycle state, persistence metadata, attention metadata, archive state

Timeline rows

Target shape once Phase 4 lands:

• agent_id
• committed seq
• committed timestamp
• canonical finalized item payload

Not part of durable history:

• raw streaming chunks
• provisional assistant text
• provisional reasoning text

Verification requirements

Every remaining phase should keep the same bar:

• npm run typecheck
• targeted tests for the touched storage/session/runtime paths
• migration/import coverage when storage authority changes
• reconnect and catch-up scenario coverage when timeline behavior changes

At minimum, timeline cutover must explicitly prove:

• fetch-after-seq
• fetch-before-seq
• restart durability
• no-gap/no-duplicate reconnect behavior

Main risks

• timeline work reintroduces provider-history replay as hidden authority
• archive behavior diverges between stored records and live in-memory agents
• explicit creation work leaves the transitional cwd fallback in place too long
• cleanup stalls after compatibility paths stop being exercised

Rule of thumb

If a new change needs to ask "what can we infer from this cwd?" for project or workspace identity, it is probably moving in the wrong direction.