# Signals Backend Implementation Plan

> **For OMP:** Create one isolated task worktree per implementation task. Run dependent tasks serially, merge each completed task branch into `feat/signals`, resolve integration conflicts on that branch, and verify the merged result before dispatching the next dependent task.

**Goal:** Build the first product-grade Signal slice: the authenticated global agent turns one or more exact user messages into one durable, Effect-modeled Signal containing immutable raw evidence and a separate processed problem statement.

**Architecture:** Keep the Signal domain pure in `@code/primitives`; keep authorization, idempotency, storage, and source resolution in Convex; keep conversation transport and the agent decision in Flue. The global Zopu agent becomes organization-scoped and receives two narrow tools: list eligible user-message sources, then create a Signal from selected message IDs and a processed problem statement. AgentOS remains downstream execution infrastructure and is not involved in Signal capture.

**Tech Stack:** Bun workspaces/catalogs, TypeScript, Effect v4, Convex, Flue, Hono, Better Auth, Vitest, convex-test.

---

## Current findings and decisions

1. `PRODUCT.md` separates **Signal**, **Candidate Work**, and **Work Unit**. This slice must not create candidate, plan, priority, run, or execution fields.
2. `TECH.md` requires explicit organization scope, exact user-message persistence, raw-evidence/agent-interpretation separation, and source links.
3. The current global chat is `zopu/main`, so every user shares one Flue instance. Flue routes are pass-through and the browser sends no identity. This must be corrected before Signals can be safe.
4. Canonical Flue `user_message` records already contain exact text plus `messageId`, `conversationId`, timestamp, and `submissionId`, but Convex stores them inside opaque JSON batches. Product commands should not parse those batches ad hoc.
5. Existing `projectSignals` is an execution event feed, not the product Signal entity. Rename it to `projectEvents` before introducing `signals`.
6. The Signal primitive will use Effect schemas and a pure Effect constructor. It will not contain a repository service or depend on Convex, Flue, Hono, AgentOS, or generated IDs.
7. A Signal is created only when the global agent decides the conversation contains a coherent problem statement. Individual user messages remain conversation evidence until then.
8. One Signal may compose multiple ordered user messages. Raw text remains per-message and exact; `problemStatement` is the agent-produced interpretation.
9. Signal creation is idempotent by organization + conversation + ordered source-message IDs so Flue retries cannot create duplicates.
10. Use a minimal organization foundation because `ownerId` is a user identifier, not the hard tenancy boundary required by the product. Do not build full organization administration in this slice.

## Domain model

```ts
interface Signal {
  readonly id: SignalId;
  readonly scope:
    | { readonly _tag: "Organization"; readonly organizationId: OrganizationId }
    | {
        readonly _tag: "Project";
        readonly organizationId: OrganizationId;
        readonly projectId: ProjectId;
      };
  readonly sourceMessages: readonly [SignalSourceMessage, ...SignalSourceMessage[]];
  readonly problemStatement: ProblemStatement;
  readonly processedBy: {
    readonly agentName: string;
    readonly agentInstanceId: string;
  };
  readonly createdAt: TimestampMs;
}

interface SignalSourceMessage {
  readonly conversationId: ConversationId;
  readonly messageId: ConversationMessageId;
  readonly rawText: string; // exact; never trim or rewrite
  readonly createdAt: TimestampMs;
}

interface ProblemStatement {
  readonly title: string;
  readonly summary: string;
  readonly desiredOutcome: string;
  readonly constraints: readonly string[];
}
```

**Invariants:**

- Every Signal has exactly one organization scope and zero or one project scope.
- `sourceMessages` is non-empty, ordered, and unique by `(conversationId, messageId)`.
- Every source is a user message from the same organization-scoped global conversation.
- `rawText` is copied server-side from the stored source message; the agent never supplies it.
- `problemStatement.title`, `summary`, and `desiredOutcome` are non-empty; constraints are structured as individual non-empty items and remain distinct from raw evidence.
- Signal does not contain candidate/work-unit/run fields.
- IDs are branded non-empty strings in primitives; adapters translate Convex/Flue IDs.
- Timestamps are non-negative integer epoch milliseconds.

## Persistence model

Add these application tables in `packages/backend/convex/schema.ts`:

- `organizations`: minimal durable tenant record with `name`, `createdBy`, `createdAt`.
- `organizationMembers`: `organizationId`, `userId` (`identity.tokenIdentifier`), `role`, `createdAt`; indexes by user and organization/user.
- `conversationMessages`: normalized user-message evidence with `organizationId`, `conversationId`, `messageId`, `submissionId?`, `clientRequestId`, `role: "user"`, exact `rawText`, `status: "admitting" | "admitted" | "failed"`, and `createdAt`; unique lookup indexes by organization/message and organization/client request.
- `signals`: `organizationId`, optional `projectId`, `sourceKey`, structured `problemStatement` (`title`, `summary`, `desiredOutcome`, `constraints`), `processedByAgentName`, `processedByAgentInstanceId`, `createdAt`; indexes by organization/createdAt, project/createdAt, and organization/sourceKey.
- `signalSources`: `signalId`, `organizationId`, `conversationId`, `messageId`, `ordinal`, `rawTextSnapshot`, `sourceCreatedAt`, optional `submissionId`; indexes by signal/ordinal and organization/message.
- `projectEvents`: the renamed current execution event feed.

Do not add lifecycle/candidate status to `signals` yet. A later Candidate component will own promotion and relationships.

---

### Task 1: Normalize the workspace dependency catalog

**Objective:** Centralize every dependency repeated across workspace manifests without forcing intentionally different platform versions into one range.

**Files:**

- Modify: `package.json`
- Modify: `apps/daemon/package.json`
- Modify: other `apps/*/package.json` and `packages/*/package.json` entries identified by the catalog audit
- Modify: `bun.lock` through `bun install`

**Steps:**

1. Add default catalog entries for shared versions that should converge: `effect`, AgentOS packages, `typescript`, `@types/bun`, `@types/react`, `heroui-native`, `vite`, `vitest`, and `convex-test`.
2. Add named catalogs for intentional platform splits, especially Node type majors and React/React Native compatibility ranges. Use `catalog:<name>` rather than silently upgrading Expo/native consumers.
3. Replace direct Effect and AgentOS pins in `apps/daemon/package.json` with `catalog:`.
4. Replace repeated direct versions in workspace manifests with their default or named catalog reference. Leave peer dependency compatibility ranges as peer ranges when they intentionally describe supported versions rather than installation versions.
5. Run `bun install` and verify the lockfile resolves the same intended platform families.

**Acceptance:**

- No repeated installed dependency version remains scattered across manifests without an explicit reason.
- Effect v4 and AgentOS resolve from the root catalog everywhere.
- Native React remains on its Expo-compatible exact family; web/TUI ranges are not accidentally collapsed.

**Verification:**

```bash
bun install
bun run check-types
```

---

### Task 2: Add the pure Effect Signal primitive

**Objective:** Establish the portable Signal vocabulary and validation rules before storage or agents depend on it.

**Files:**

- Create: `packages/primitives/src/signal.ts`
- Create: `packages/primitives/src/signal.test.ts`
- Modify: `packages/primitives/src/index.ts`
- Modify: `packages/primitives/package.json`

**Implementation:**

- Define branded schemas for `SignalId`, `OrganizationId`, `ProjectId`, `ConversationId`, `ConversationMessageId`, and `TimestampMs`.
- Define `SignalScope`, `SignalSourceMessage`, `ProblemStatement`, `ProcessedByAgent`, `SignalInput`, and `Signal` with `Schema.Struct` / discriminated unions.
- Add `SignalValidationError` with `Schema.TaggedErrorClass`.
- Add `composeSignal = Effect.fn("Signal.compose")` that decodes input, rejects duplicate/out-of-order sources, preserves raw whitespace exactly, and returns a validated Signal.
- Export the module from the root and an explicit `./signal` package subpath.
- Add a package `test` script using cataloged Vitest.

**Tests:**

- Accept organization and project scopes.
- Reject empty IDs, empty source list, empty problem-statement title/summary/desired outcome, empty constraint items, negative/fractional timestamps, duplicate source IDs, mixed conversations, and out-of-order timestamps.
- Prove raw leading/trailing whitespace and source ordering survive unchanged.
- Prove candidate/work-unit/run fields are absent from the decoded result.

**Verification:**

```bash
bun run --cwd packages/primitives test
bun run --cwd packages/primitives check-types
bunx ultracite check packages/primitives/src/signal.ts packages/primitives/src/signal.test.ts packages/primitives/src/index.ts
```

---

### Task 3: Establish minimal organization tenancy

**Objective:** Give every durable Signal and global conversation a real tenant boundary instead of reusing a user ID or `main`.

**Files:**

- Modify: `packages/backend/convex/schema.ts`
- Modify: `packages/backend/convex/authz.ts`
- Create: `packages/backend/convex/organizations.ts`
- Create: `packages/backend/convex/organizations.test.ts`

**Implementation:**

- Add `organizations` and `organizationMembers` tables.
- Add `ensurePersonalOrganization`, idempotently creating one personal organization and owner membership for the authenticated user.
- Add `getCurrent` and `requireOrganizationMember` helpers.
- Keep full organization invitations, multiple active organizations, roles beyond owner/member, and organization settings out of scope.
- Ensure every query/mutation denies access without both authenticated identity and membership.

**Tests:**

- First ensure creates one organization and owner membership.
- Repeated ensure returns the same organization.
- A second identity cannot read or mutate the first identity's organization.

**Verification:**

```bash
bun run --cwd packages/backend test -- organizations
bun run --cwd packages/backend check-types
```

---

### Task 4: Rename execution signals to project events

**Objective:** Remove the naming collision before creating the product `signals` table.

**Files:**

- Modify: `packages/backend/convex/schema.ts`
- Modify: `packages/backend/convex/projects.ts`
- Modify: `packages/backend/convex/projectIssues.ts`
- Modify: `packages/backend/convex/agentWorkspace.ts`
- Modify: `packages/backend/convex/artifactModel.ts`
- Modify: `packages/backend/convex/README.md`
- Optional migration file only if existing rows are present: `packages/backend/convex/migrations.ts`

**Steps:**

1. Inspect the active Convex deployment count for `projectSignals` before changing schema.
2. If empty, cleanly rename the table and call sites to `projectEvents`.
3. If non-empty, add a one-off internal migration that copies rows with preserved relations/timestamps, verify source/destination counts, then remove the old schema/table references. Do not leave a compatibility alias.
4. Update `signals.md` seed copy to say “project events,” not product Signals.

**Acceptance:**

- `projectSignals` no longer exists in source or schema.
- Existing project event behavior remains unchanged.
- Product `signals` can be introduced without semantic ambiguity.

**Verification:**

```bash
bun run --cwd packages/backend check-types
```

Exercise project connect, issue create/start, artifact update, and agent status once and confirm `projectEvents` receives the same events.

---

### Task 5: Normalize authenticated global user messages

**Objective:** Persist exact tenant-scoped user evidence at the conversation ingress before the agent can create Signals.

**Files:**

- Modify: `packages/backend/convex/schema.ts`
- Create: `packages/backend/convex/conversationMessages.ts`
- Create: `packages/backend/convex/conversationMessages.test.ts`
- Modify: `packages/agents/src/agents/zopu.ts`
- Create: `packages/agents/src/auth.ts`
- Modify: `packages/agents/package.json`
- Modify: `apps/web/src/root.tsx`
- Modify: `apps/web/src/lib/chat/constants.ts`
- Modify: `apps/web/src/hooks/chat/use-chat-agent.ts`
- Modify: `packages/auth/src/web/index.ts` only if the access-token helper is not already exported

**Ingress sequence:**

1. Web obtains the Better Auth/Convex access token with the installed `fetchAccessToken` API.
2. The custom Flue fetch adds `Authorization: Bearer <token>` and a stable `x-zopu-request-id` to agent POSTs.
3. Zopu route middleware requires the bearer token, sets it on a short-lived `ConvexHttpClient`, resolves current organization membership, and rejects an agent instance ID that is not that organization ID.
4. Before `next()`, middleware clones and validates the direct payload and calls `conversationMessages.beginUserMessage` with exact text and request ID.
5. After Flue returns its admission receipt, middleware patches `submissionId` and status `admitted`; on failure it marks the message `failed` and rethrows/returns the original failure.
6. The web global agent ID becomes the current organization ID rather than `main`.

**Important constraints:**

- Do not trust organization ID, raw text, timestamp, or source role from an agent tool.
- Do not parse `flueConversationBatches.recordsJson` for product commands.
- Do not persist auth tokens.
- Keep failed admissions as evidence with explicit status; do not silently delete them.
- Ensure GET/SSE observation routes use the same organization authorization.

**Tests:**

- Unauthorized Flue route is rejected.
- Organization A cannot use or observe organization B's agent ID.
- Duplicate request ID is idempotent.
- Exact whitespace/casing is stored.
- Success records submission ID; failure records failed status.

**Verification:**

- Send one message containing leading/trailing whitespace through the running web app.
- Confirm the agent replies.
- Query Convex and confirm the normalized user message has exact raw text, organization scope, conversation/message identity, request ID, and admitted submission ID.
- Sign in as another identity and confirm it cannot observe the first conversation.

---

### Task 6: Persist Signals atomically in Convex

**Objective:** Resolve selected message IDs server-side, run the Effect constructor, and atomically insert one idempotent Signal plus ordered source snapshots.

**Files:**

- Modify: `packages/backend/convex/schema.ts`
- Create: `packages/backend/convex/signals.ts`
- Create: `packages/backend/convex/signals.test.ts`
- Modify: `packages/backend/package.json`

**Interface:**

```ts
createFromMessages({
  organizationId,
  projectId?,
  conversationId,
  messageIds,
  problemStatement: { title, summary, desiredOutcome, constraints },
  processedBy: { agentName, agentInstanceId },
}) -> signalId
```

**Implementation:**

- Expose an agent-token-gated mutation for Zopu and authenticated read queries for product clients.
- Resolve every message from `conversationMessages` under the same organization and conversation.
- Require role `user`, status `admitted`, non-empty ordered selection, and optional project ownership by the same organization.
- Copy raw text/timestamps/submission IDs into `signalSources`; never accept snapshots from the agent.
- Build `sourceKey` deterministically from organization, conversation, and ordered message IDs.
- Run `composeSignal` from `@code/primitives/signal` before writes.
- In one Convex mutation, return an existing row for the same source key or insert `signals` and all `signalSources` in ordinal order.
- Add bounded `list` and `get` queries that enforce membership and return the composed source records.

**Tests:**

- Creates one global Signal from one user message.
- Composes several ordered user messages into one Signal.
- Preserves exact raw snapshots while storing a separate structured problem statement.
- Rejects cross-organization, cross-conversation, assistant, failed, missing, duplicated, and unauthorized project messages.
- Repeating the same source set returns the existing Signal without duplicate source rows.

**Verification:**

```bash
bun run --cwd packages/backend test -- signals
bun run --cwd packages/backend check-types
bunx ultracite check packages/backend/convex/schema.ts packages/backend/convex/signals.ts packages/backend/convex/signals.test.ts
```

---

### Task 7: Let Zopu decide and create Signals

**Objective:** Give the global agent a narrow, validated mechanism to inspect eligible user evidence and create a Signal when the problem statement is coherent.

**Files:**

- Create: `packages/agents/src/tools/signals.ts`
- Modify: `packages/agents/src/agents/zopu.ts`
- Modify: `packages/agents/package.json`
- Create: `packages/agents/scripts/signal-smoke.ts`

**Tools:**

1. `list_signal_source_messages`
   - No free tenant input; organization comes from the authenticated agent instance ID.
   - Returns a bounded list of admitted user messages not already used by a Signal, including message ID, exact text, and timestamp.
2. `create_signal`
   - Input: ordered message IDs, structured problem statement (`title`, `summary`, `desiredOutcome`, `constraints`), optional project ID.
   - Calls the Convex `createFromMessages` mutation with `agentName: "zopu"` and the current organization-scoped instance ID.
   - Returns only the Signal ID and source count.

**Agent instruction:**

- Continue normal planning conversation until the user's instructions form a coherent problem statement.
- Do not create a Signal for greetings, exploration without a concrete problem, or every individual message.
- When ready, call `list_signal_source_messages`, select only the messages composing the problem, then call `create_signal` once.
- Tell the user the captured problem-statement title, summary, desired outcome, and constraints.
- Do not create Candidate Work or start AgentOS in this slice.

**Retry safety:**

- Flue may replay tool calls after interruption. Convex source-key idempotency is authoritative.
- The tool must surface domain errors; it must not swallow or convert them into fake success.

**Verification:**

Run the focused smoke script and a live browser scenario:

1. Send three messages: an initial problem, a clarification, and an explicit final constraint.
2. Observe Zopu create exactly one Signal after the statement is coherent.
3. Confirm the Signal contains all selected raw messages in order and one structured problem statement with title, summary, desired outcome, and constraints.
4. Restart the agent service/retry the final turn and confirm no duplicate Signal is inserted.
5. Confirm no AgentOS actor, candidate, work unit, issue, or run is created.

```bash
bun run --cwd packages/agents signal:smoke
bun run --cwd packages/agents check-types
bun run --cwd apps/web check-types
bunx ultracite check packages/agents/src/tools/signals.ts packages/agents/src/agents/zopu.ts packages/agents/scripts/signal-smoke.ts apps/web/src/root.tsx apps/web/src/lib/chat/constants.ts apps/web/src/hooks/chat/use-chat-agent.ts
```

---

### Task 8: Final verification and cleanup

**Objective:** Prove the complete Signal slice works and remove temporary migration/scaffolding.

**Steps:**

1. Run all focused tests and smoke scenarios from Tasks 2–7.
2. Run Ultracite fix if necessary, then check every changed source/script/agent file.
3. Run affected package type checks: primitives, backend, agents, auth if touched, and web.
4. Run root `bun run check` and report unrelated pre-existing failures separately.
5. Remove a one-off `projectSignals` migration only after row-count verification and successful cutover, unless the repository convention keeps completed migrations.
6. Update existing backend/agent documentation only where names or commands became stale; do not add speculative docs.

**Final acceptance criteria:**

- Global conversations are organization-scoped and authenticated; `zopu/main` is gone.
- Exact user text is stored before/with Flue admission and cannot cross organizations.
- Zopu can intentionally compose one or more user messages into exactly one durable Signal.
- Signal raw evidence and the structured processed problem statement are separate and queryable.
- Source provenance is immutable and ordered.
- Retry produces no duplicate Signal.
- Existing project execution events still work under `projectEvents`.
- Signal creation does not create Candidate Work, Work Unit, run, or AgentOS resources.
- Effect v4 and AgentOS dependency versions come from workspace catalogs.
- Focused tests, package type checks, runtime smoke, and root check are reported with evidence.

## Risks and deliberate tradeoffs

- **Minimal organization bootstrap:** This adds only the tenancy needed for Signals. Multi-org switching, invites, billing, and policy are separate components.
- **Normalized user evidence plus Flue transcript:** There are temporarily two durable representations. Flue remains the canonical conversational transcript; `conversationMessages` is the application-domain evidence index required for authorization and provenance. Idempotent request IDs and submission links keep them correlated.
- **No automatic Signal for every message:** This follows the product thesis and avoids task noise. The agent explicitly decides when the instructions are coherent.
- **No separate Flow agent yet:** Zopu performs the first extraction using narrow tools. A future Flow Agent can adopt the same Convex command without changing the Signal primitive.
- **No AgentOS here:** AgentOS begins only after future Candidate/Work Unit readiness and scheduling.
- **No Signal status model yet:** Candidate attachment, knowledge capture, dismissal, and work promotion belong to later components. Adding them now would pre-design unimplemented workflows.
