zopu-code/docs/2026-07-23_140325-signals-backend.md

Signals Backend Implementation Plan

For OMP: Create one isolated task worktree per implementation task. Run dependent tasks serially, merge each completed task branch into feat/signals, resolve integration conflicts on that branch, and verify the merged result before dispatching the next dependent task.

Goal: Build the first product-grade Signal slice: the authenticated global agent turns one or more exact user messages into one durable, Effect-modeled Signal containing immutable raw evidence and a separate processed problem statement.

Architecture: Keep the Signal domain pure in @code/primitives; keep authorization, idempotency, storage, and source resolution in Convex; keep conversation transport and the agent decision in Flue. The global Zopu agent becomes organization-scoped and receives two narrow tools: list eligible user-message sources, then create a Signal from selected message IDs and a processed problem statement. AgentOS remains downstream execution infrastructure and is not involved in Signal capture.

Tech Stack: Bun workspaces/catalogs, TypeScript, Effect v4, Convex, Flue, Hono, Better Auth, Vitest, convex-test.

---

Current findings and decisions

1. PRODUCT.md separates Signal, Candidate Work, and Work Unit. This slice must not create candidate, plan, priority, run, or execution fields.
2. TECH.md requires explicit organization scope, exact user-message persistence, raw-evidence/agent-interpretation separation, and source links.
3. The current global chat is zopu/main, so every user shares one Flue instance. Flue routes are pass-through and the browser sends no identity. This must be corrected before Signals can be safe.
4. Canonical Flue user_message records already contain exact text plus messageId, conversationId, timestamp, and submissionId, but Convex stores them inside opaque JSON batches. Product commands should not parse those batches ad hoc.
5. Existing projectSignals is an execution event feed, not the product Signal entity. Rename it to projectEvents before introducing signals.
6. The Signal primitive will use Effect schemas and a pure Effect constructor. It will not contain a repository service or depend on Convex, Flue, Hono, AgentOS, or generated IDs.
7. A Signal is created only when the global agent decides the conversation contains a coherent problem statement. Individual user messages remain conversation evidence until then.
8. One Signal may compose multiple ordered user messages. Raw text remains per-message and exact; problemStatement is the agent-produced interpretation.
9. Signal creation is idempotent by organization + conversation + ordered source-message IDs so Flue retries cannot create duplicates.
10. Use a minimal organization foundation because ownerId is a user identifier, not the hard tenancy boundary required by the product. Do not build full organization administration in this slice.

Domain model

interface Signal {
  readonly id: SignalId;
  readonly scope:
    | { readonly _tag: "Organization"; readonly organizationId: OrganizationId }
    | {
        readonly _tag: "Project";
        readonly organizationId: OrganizationId;
        readonly projectId: ProjectId;
      };
  readonly sourceMessages: readonly [SignalSourceMessage, ...SignalSourceMessage[]];
  readonly problemStatement: ProblemStatement;
  readonly processedBy: {
    readonly agentName: string;
    readonly agentInstanceId: string;
  };
  readonly createdAt: TimestampMs;
}

interface SignalSourceMessage {
  readonly conversationId: ConversationId;
  readonly messageId: ConversationMessageId;
  readonly rawText: string; // exact; never trim or rewrite
  readonly createdAt: TimestampMs;
}

interface ProblemStatement {
  readonly title: string;
  readonly summary: string;
  readonly desiredOutcome: string;
  readonly constraints: readonly string[];
}

Invariants:

• Every Signal has exactly one organization scope and zero or one project scope.
• sourceMessages is non-empty, ordered, and unique by (conversationId, messageId).
• Every source is a user message from the same organization-scoped global conversation.
• rawText is copied server-side from the stored source message; the agent never supplies it.
• problemStatement.title, summary, and desiredOutcome are non-empty; constraints are structured as individual non-empty items and remain distinct from raw evidence.
• Signal does not contain candidate/work-unit/run fields.
• IDs are branded non-empty strings in primitives; adapters translate Convex/Flue IDs.
• Timestamps are non-negative integer epoch milliseconds.

Persistence model

Add these application tables in packages/backend/convex/schema.ts:

• organizations: minimal durable tenant record with name, createdBy, createdAt.
• organizationMembers: organizationId, userId (identity.tokenIdentifier), role, createdAt; indexes by user and organization/user.
• conversationMessages: normalized user-message evidence with organizationId, conversationId, messageId, submissionId?, clientRequestId, role: "user", exact rawText, status: "admitting" | "admitted" | "failed", and createdAt; unique lookup indexes by organization/message and organization/client request.
• signals: organizationId, optional projectId, sourceKey, structured problemStatement (title, summary, desiredOutcome, constraints), processedByAgentName, processedByAgentInstanceId, createdAt; indexes by organization/createdAt, project/createdAt, and organization/sourceKey.
• signalSources: signalId, organizationId, conversationId, messageId, ordinal, rawTextSnapshot, sourceCreatedAt, optional submissionId; indexes by signal/ordinal and organization/message.
• projectEvents: the renamed current execution event feed.

Do not add lifecycle/candidate status to signals yet. A later Candidate component will own promotion and relationships.

---

Task 1: Normalize the workspace dependency catalog

Objective: Centralize every dependency repeated across workspace manifests without forcing intentionally different platform versions into one range.

Files:

• Modify: package.json
• Modify: apps/daemon/package.json
• Modify: other apps/*/package.json and packages/*/package.json entries identified by the catalog audit
• Modify: bun.lock through bun install

Steps:

1. Add default catalog entries for shared versions that should converge: effect, AgentOS packages, typescript, @types/bun, @types/react, heroui-native, vite, vitest, and convex-test.
2. Add named catalogs for intentional platform splits, especially Node type majors and React/React Native compatibility ranges. Use catalog:<name> rather than silently upgrading Expo/native consumers.
3. Replace direct Effect and AgentOS pins in apps/daemon/package.json with catalog:.
4. Replace repeated direct versions in workspace manifests with their default or named catalog reference. Leave peer dependency compatibility ranges as peer ranges when they intentionally describe supported versions rather than installation versions.
5. Run bun install and verify the lockfile resolves the same intended platform families.

Acceptance:

• No repeated installed dependency version remains scattered across manifests without an explicit reason.
• Effect v4 and AgentOS resolve from the root catalog everywhere.
• Native React remains on its Expo-compatible exact family; web/TUI ranges are not accidentally collapsed.

Verification:

bun install
bun run check-types

---

Task 2: Add the pure Effect Signal primitive

Objective: Establish the portable Signal vocabulary and validation rules before storage or agents depend on it.

Files:

• Create: packages/primitives/src/signal.ts
• Create: packages/primitives/src/signal.test.ts
• Modify: packages/primitives/src/index.ts
• Modify: packages/primitives/package.json

Implementation:

• Define branded schemas for SignalId, OrganizationId, ProjectId, ConversationId, ConversationMessageId, and TimestampMs.
• Define SignalScope, SignalSourceMessage, ProblemStatement, ProcessedByAgent, SignalInput, and Signal with Schema.Struct / discriminated unions.
• Add SignalValidationError with Schema.TaggedErrorClass.
• Add composeSignal = Effect.fn("Signal.compose") that decodes input, rejects duplicate/out-of-order sources, preserves raw whitespace exactly, and returns a validated Signal.
• Export the module from the root and an explicit ./signal package subpath.
• Add a package test script using cataloged Vitest.

Tests:

• Accept organization and project scopes.
• Reject empty IDs, empty source list, empty problem-statement title/summary/desired outcome, empty constraint items, negative/fractional timestamps, duplicate source IDs, mixed conversations, and out-of-order timestamps.
• Prove raw leading/trailing whitespace and source ordering survive unchanged.
• Prove candidate/work-unit/run fields are absent from the decoded result.

Verification:

bun run --cwd packages/primitives test
bun run --cwd packages/primitives check-types
bunx ultracite check packages/primitives/src/signal.ts packages/primitives/src/signal.test.ts packages/primitives/src/index.ts

---

Task 3: Establish minimal organization tenancy

Objective: Give every durable Signal and global conversation a real tenant boundary instead of reusing a user ID or main.

Files:

• Modify: packages/backend/convex/schema.ts
• Modify: packages/backend/convex/authz.ts
• Create: packages/backend/convex/organizations.ts
• Create: packages/backend/convex/organizations.test.ts

Implementation:

• Add organizations and organizationMembers tables.
• Add ensurePersonalOrganization, idempotently creating one personal organization and owner membership for the authenticated user.
• Add getCurrent and requireOrganizationMember helpers.
• Keep full organization invitations, multiple active organizations, roles beyond owner/member, and organization settings out of scope.
• Ensure every query/mutation denies access without both authenticated identity and membership.

Tests:

• First ensure creates one organization and owner membership.
• Repeated ensure returns the same organization.
• A second identity cannot read or mutate the first identity's organization.

Verification:

bun run --cwd packages/backend test -- organizations
bun run --cwd packages/backend check-types

---

Task 4: Rename execution signals to project events

Objective: Remove the naming collision before creating the product signals table.

Files:

• Modify: packages/backend/convex/schema.ts
• Modify: packages/backend/convex/projects.ts
• Modify: packages/backend/convex/projectIssues.ts
• Modify: packages/backend/convex/agentWorkspace.ts
• Modify: packages/backend/convex/artifactModel.ts
• Modify: packages/backend/convex/README.md
• Optional migration file only if existing rows are present: packages/backend/convex/migrations.ts

Steps:

1. Inspect the active Convex deployment count for projectSignals before changing schema.
2. If empty, cleanly rename the table and call sites to projectEvents.
3. If non-empty, add a one-off internal migration that copies rows with preserved relations/timestamps, verify source/destination counts, then remove the old schema/table references. Do not leave a compatibility alias.
4. Update signals.md seed copy to say “project events,” not product Signals.

Acceptance:

• projectSignals no longer exists in source or schema.
• Existing project event behavior remains unchanged.
• Product signals can be introduced without semantic ambiguity.

Verification:

bun run --cwd packages/backend check-types

Exercise project connect, issue create/start, artifact update, and agent status once and confirm projectEvents receives the same events.

---

Task 5: Normalize authenticated global user messages

Objective: Persist exact tenant-scoped user evidence at the conversation ingress before the agent can create Signals.

Files:

• Modify: packages/backend/convex/schema.ts
• Create: packages/backend/convex/conversationMessages.ts
• Create: packages/backend/convex/conversationMessages.test.ts
• Modify: packages/agents/src/agents/zopu.ts
• Create: packages/agents/src/auth.ts
• Modify: packages/agents/package.json
• Modify: apps/web/src/root.tsx
• Modify: apps/web/src/lib/chat/constants.ts
• Modify: apps/web/src/hooks/chat/use-chat-agent.ts
• Modify: packages/auth/src/web/index.ts only if the access-token helper is not already exported

Ingress sequence:

1. Web obtains the Better Auth/Convex access token with the installed fetchAccessToken API.
2. The custom Flue fetch adds Authorization: Bearer <token> and a stable x-zopu-request-id to agent POSTs.
3. Zopu route middleware requires the bearer token, sets it on a short-lived ConvexHttpClient, resolves current organization membership, and rejects an agent instance ID that is not that organization ID.
4. Before next(), middleware clones and validates the direct payload and calls conversationMessages.beginUserMessage with exact text and request ID.
5. After Flue returns its admission receipt, middleware patches submissionId and status admitted; on failure it marks the message failed and rethrows/returns the original failure.
6. The web global agent ID becomes the current organization ID rather than main.

Important constraints:

• Do not trust organization ID, raw text, timestamp, or source role from an agent tool.
• Do not parse flueConversationBatches.recordsJson for product commands.
• Do not persist auth tokens.
• Keep failed admissions as evidence with explicit status; do not silently delete them.
• Ensure GET/SSE observation routes use the same organization authorization.

Tests:

• Unauthorized Flue route is rejected.
• Organization A cannot use or observe organization B's agent ID.
• Duplicate request ID is idempotent.
• Exact whitespace/casing is stored.
• Success records submission ID; failure records failed status.

Verification:

• Send one message containing leading/trailing whitespace through the running web app.
• Confirm the agent replies.
• Query Convex and confirm the normalized user message has exact raw text, organization scope, conversation/message identity, request ID, and admitted submission ID.
• Sign in as another identity and confirm it cannot observe the first conversation.

---

Task 6: Persist Signals atomically in Convex

Objective: Resolve selected message IDs server-side, run the Effect constructor, and atomically insert one idempotent Signal plus ordered source snapshots.

Files:

• Modify: packages/backend/convex/schema.ts
• Create: packages/backend/convex/signals.ts
• Create: packages/backend/convex/signals.test.ts
• Modify: packages/backend/package.json

Interface:

createFromMessages({
  organizationId,
  projectId?,
  conversationId,
  messageIds,
  problemStatement: { title, summary, desiredOutcome, constraints },
  processedBy: { agentName, agentInstanceId },
}) -> signalId

Implementation:

• Expose an agent-token-gated mutation for Zopu and authenticated read queries for product clients.
• Resolve every message from conversationMessages under the same organization and conversation.
• Require role user, status admitted, non-empty ordered selection, and optional project ownership by the same organization.
• Copy raw text/timestamps/submission IDs into signalSources; never accept snapshots from the agent.
• Build sourceKey deterministically from organization, conversation, and ordered message IDs.
• Run composeSignal from @code/primitives/signal before writes.
• In one Convex mutation, return an existing row for the same source key or insert signals and all signalSources in ordinal order.
• Add bounded list and get queries that enforce membership and return the composed source records.

Tests:

• Creates one global Signal from one user message.
• Composes several ordered user messages into one Signal.
• Preserves exact raw snapshots while storing a separate structured problem statement.
• Rejects cross-organization, cross-conversation, assistant, failed, missing, duplicated, and unauthorized project messages.
• Repeating the same source set returns the existing Signal without duplicate source rows.

Verification:

bun run --cwd packages/backend test -- signals
bun run --cwd packages/backend check-types
bunx ultracite check packages/backend/convex/schema.ts packages/backend/convex/signals.ts packages/backend/convex/signals.test.ts

---

Task 7: Let Zopu decide and create Signals

Objective: Give the global agent a narrow, validated mechanism to inspect eligible user evidence and create a Signal when the problem statement is coherent.

Files:

• Create: packages/agents/src/tools/signals.ts
• Modify: packages/agents/src/agents/zopu.ts
• Modify: packages/agents/package.json
• Create: packages/agents/scripts/signal-smoke.ts

Tools:

1. list_signal_source_messages
   • No free tenant input; organization comes from the authenticated agent instance ID.
   • Returns a bounded list of admitted user messages not already used by a Signal, including message ID, exact text, and timestamp.
2. create_signal
   • Input: ordered message IDs, structured problem statement (title, summary, desiredOutcome, constraints), optional project ID.
   • Calls the Convex createFromMessages mutation with agentName: "zopu" and the current organization-scoped instance ID.
   • Returns only the Signal ID and source count.

Agent instruction:

• Continue normal planning conversation until the user's instructions form a coherent problem statement.
• Do not create a Signal for greetings, exploration without a concrete problem, or every individual message.
• When ready, call list_signal_source_messages, select only the messages composing the problem, then call create_signal once.
• Tell the user the captured problem-statement title, summary, desired outcome, and constraints.
• Do not create Candidate Work or start AgentOS in this slice.

Retry safety:

• Flue may replay tool calls after interruption. Convex source-key idempotency is authoritative.
• The tool must surface domain errors; it must not swallow or convert them into fake success.

Verification:

Run the focused smoke script and a live browser scenario:

1. Send three messages: an initial problem, a clarification, and an explicit final constraint.
2. Observe Zopu create exactly one Signal after the statement is coherent.
3. Confirm the Signal contains all selected raw messages in order and one structured problem statement with title, summary, desired outcome, and constraints.
4. Restart the agent service/retry the final turn and confirm no duplicate Signal is inserted.
5. Confirm no AgentOS actor, candidate, work unit, issue, or run is created.

bun run --cwd packages/agents signal:smoke
bun run --cwd packages/agents check-types
bun run --cwd apps/web check-types
bunx ultracite check packages/agents/src/tools/signals.ts packages/agents/src/agents/zopu.ts packages/agents/scripts/signal-smoke.ts apps/web/src/root.tsx apps/web/src/lib/chat/constants.ts apps/web/src/hooks/chat/use-chat-agent.ts

---

Task 8: Final verification and cleanup

Objective: Prove the complete Signal slice works and remove temporary migration/scaffolding.

Steps:

1. Run all focused tests and smoke scenarios from Tasks 2–7.
2. Run Ultracite fix if necessary, then check every changed source/script/agent file.
3. Run affected package type checks: primitives, backend, agents, auth if touched, and web.
4. Run root bun run check and report unrelated pre-existing failures separately.
5. Remove a one-off projectSignals migration only after row-count verification and successful cutover, unless the repository convention keeps completed migrations.
6. Update existing backend/agent documentation only where names or commands became stale; do not add speculative docs.

Final acceptance criteria:

• Global conversations are organization-scoped and authenticated; zopu/main is gone.
• Exact user text is stored before/with Flue admission and cannot cross organizations.
• Zopu can intentionally compose one or more user messages into exactly one durable Signal.
• Signal raw evidence and the structured processed problem statement are separate and queryable.
• Source provenance is immutable and ordered.
• Retry produces no duplicate Signal.
• Existing project execution events still work under projectEvents.
• Signal creation does not create Candidate Work, Work Unit, run, or AgentOS resources.
• Effect v4 and AgentOS dependency versions come from workspace catalogs.
• Focused tests, package type checks, runtime smoke, and root check are reported with evidence.

Risks and deliberate tradeoffs

• Minimal organization bootstrap: This adds only the tenancy needed for Signals. Multi-org switching, invites, billing, and policy are separate components.
• Normalized user evidence plus Flue transcript: There are temporarily two durable representations. Flue remains the canonical conversational transcript; conversationMessages is the application-domain evidence index required for authorization and provenance. Idempotent request IDs and submission links keep them correlated.
• No automatic Signal for every message: This follows the product thesis and avoids task noise. The agent explicitly decides when the instructions are coherent.
• No separate Flow agent yet: Zopu performs the first extraction using narrow tools. A future Flow Agent can adopt the same Convex command without changing the Signal primitive.
• No AgentOS here: AgentOS begins only after future Candidate/Work Unit readiness and scheduling.
• No Signal status model yet: Candidate attachment, knowledge capture, dismissal, and work promotion belong to later components. Adding them now would pre-design unimplemented workflows.