zopu-code/docs/evaluation.md

# Zopu Work OS — Evaluation and Improvement

> **Purpose:** measure whether Zopu produces better software and improve agents, Kits, tools, prompts, and workflow without regressions.  
> **Principle:** no autonomy increase without evidence.

## 1. Evaluation targets

Evaluate five layers separately:

```text
A. Intent quality
B. Design quality
C. Execution reliability
D. Software quality
E. Product outcome
```

A fast agent that generates more code but increases review/rework is a regression.

## 2. Core metrics

### Intent

- Signal precision/recall on labeled messages;
- duplicate/incorrect Work rate;
- acceptance-criteria correction rate;
- unresolved ambiguity discovered after implementation.

### Design

- human design revision rate;
- implementation deviation rate;
- missed impacted components;
- slice replanning rate;
- architecture-review issues discovered after coding.

### Execution

- terminal Work percentage;
- stale-running count;
- attempt success/retry/failure distribution;
- restart recovery success;
- duplicate side effects;
- cancellation cleanup;
- time/cost per verified slice.

### Software quality

- verification pass rate on first candidate;
- escaped defect/regression rate;
- review changes requested;
- test weakening incidents;
- security/maintainability findings;
- integration-only failures;
- post-merge rollback/hotfix rate.

### Human experience

- attention minutes per verified outcome;
- blocking questions per Work;
- time to understand review package;
- percentage of recommendations accepted;
- user override/correction rate.

### Outcome

- achieved/partial/failed/inconclusive distribution;
- time from merge to result;
- original success signal movement;
- follow-up incident rate.

## 3. Evaluation corpus

Maintain a versioned fixture corpus.

### Message/Signal fixtures

```text
casual conversation
clear actionable request
multiple intents
duplicate request
new evidence for existing Work
conflicting requirement
incident/error Signal
```

### Software Work fixtures

```text
small isolated bug
API behavior addition
frontend/backend vertical flow
cross-package refactor
auth/permission change
database migration
parallel-slice conflict
verification repair
human question/resume
release regression
```

Each fixture includes:

```text
repository/base SHA
input Signals
expected Work Definition properties
risk lane
expected impacted areas
required behavior/evidence
forbidden outcomes
human gold review notes
```

## 4. Evaluation modes

### Deterministic unit/property evaluation

For:

- schemas;
- state transitions;
- dependency readiness;
- idempotency;
- budgets/retries;
- approval invalidation;
- actor recovery.

Must run on every change.

### Adapter contract evaluation

Run the same suite against fake and live:

```text
HarnessRuntime
SandboxRuntime
SourceControl
ArtifactStore
PreviewRuntime
```

### Agent offline evaluation

Replay fixture inputs against candidate agent/Kit versions. Store all outputs and compare with current production baseline.

### End-to-end sandbox evaluation

Run real repository tasks in isolated environments, verify exact candidate, and score review quality.

### Dogfood/online evaluation

Observe real Work with human decisions, merge outcomes, incidents, and post-release results.

## 5. Scoring rubric

Score each candidate run 0–3 per dimension.

| Score | Meaning |
|---|---|
| 0 | unsafe/incorrect/missing |
| 1 | substantial human repair required |
| 2 | acceptable with minor correction |
| 3 | correct, complete, reviewable |

Dimensions:

```text
intent fidelity
scope control
design correctness
slice quality
implementation correctness
test quality
maintainability
security/reliability
evidence completeness
review clarity
result achievement
```

Critical safety/invariant violation is automatic failure regardless of aggregate score.

## 6. Golden invariants

Automated assertions:

```text
no duplicate Signal/Work/PR/deployment
no Work stuck active beyond lease/timeout
no verification without SHA/environment
no PR head different from verified SHA
no stale approval accepted
no cross-worktree mutation
no unscoped persistent credential
no builder-only completion verdict
no retry beyond policy
no silent canonical-knowledge mutation
```

## 7. Failure taxonomy

Classify failures consistently:

```text
Intent
├── missed Signal
├── wrong Work
├── scope drift
└── unresolved ambiguity

Design
├── missed impact
├── wrong boundary
├── invalid slice
└── hidden trade-off

Execution
├── model failure
├── harness failure
├── sandbox failure
├── tool failure
├── timeout/cancellation
└── recovery/idempotency failure

Verification
├── missing check
├── flaky check
├── false pass
├── false failure
└── wrong revision/environment

Delivery
├── integration conflict
├── wrong PR revision
├── release failure
└── rollback failure

Outcome
├── user problem remains
├── regression
└── inconclusive observation
```

Every failed Work records one primary and optional contributing categories.

## 8. Experiment protocol

Any change to agent, prompt, model, Kit, tool, runtime policy, or Resolver heuristic requires:

```text
hypothesis
target metric
candidate version
baseline version
fixture set
budget
results
regressions
decision
```

Recommended sequence:

```text
offline corpus
→ shadow/dry run
→ small dogfood cohort
→ policy-controlled rollout
```

Do not promote based on one impressive example.

## 9. Agent/Kit promotion gate

A candidate may replace baseline only when:

- golden invariants pass;
- no critical regression;
- task success does not decrease materially;
- review/rework improves or remains acceptable;
- compute/time increase is justified;
- failure mode and rollback are known;
- version can be selected/reverted independently.

Record promotion as a durable decision.

## 10. Verifier evaluation

Verifiers themselves need evaluation.

Test:

- whether injected defects are caught;
- false-positive rate;
- whether checks bind exact candidate;
- whether design deviations are identified;
- whether evidence is concise and actionable;
- whether repair instructions reproduce the failure.

Use mutation/injected-defect suites where practical.

## 11. Human review calibration

Periodically compare:

```text
agent/verifier verdict
human reviewer verdict
post-merge outcome
```

Track disagreement categories. A reviewer model agreeing with another model is not ground truth.

## 12. Post-release feedback

For completed Work, schedule/perform observation using defined success signals.

Convert:

- incidents;
- support complaints;
- failed health checks;
- rollback;
- unexpected metrics;
- manual rework

into linked Signals and evaluation records.

## 13. Improvement outputs

Evaluation may produce proposals for:

```text
Work Definition rule
Design Packet template
slice heuristic
verification recipe
Kit/role/tool
runtime policy
repository knowledge
UI review package
```

Each proposal states evidence, expected benefit, risk, and rollback.

## 14. Dashboard minimum

Internal dogfood dashboard:

```text
active/stale Work
terminal outcome distribution
first-pass verification
retry/replan causes
human wait/attention
duplicate side effects
review changes requested
post-merge failures
cost/time per verified Work
agent/Kit version comparison
```

Do not collapse quality into one vanity score.

## 15. Initial benchmark gate

Before v0 dogfood, pass:

```text
✓ actionable vs casual Signal fixtures
✓ duplicate processing
✓ definition/design version invalidation
✓ fake Resolver crash recovery
✓ sandbox cancellation
✓ verification repair
✓ exact-SHA PR invariant
✓ human question/resume
✓ duplicate callback/PR prevention
```

Before increasing autonomy, add:

```text
✓ integration conflict
✓ auth/permission fixture
✓ migration/rollback fixture
✓ post-release regression fixture
```

## 16. Continuous improvement loop

```text
Run Work
→ capture structured outcome
→ classify failures
→ aggregate patterns
→ propose one bounded change
→ evaluate against corpus
→ dogfood
→ promote or revert
```

Improve the weakest stage, not the most fashionable model.