zopu-code/docs/evaluation.md

Zopu Work OS — Evaluation and Improvement

Purpose: measure whether Zopu produces better software and improve agents, Kits, tools, prompts, and workflow without regressions.
Principle: no autonomy increase without evidence.

1. Evaluation targets

Evaluate five layers separately:

A. Intent quality
B. Design quality
C. Execution reliability
D. Software quality
E. Product outcome

A fast agent that generates more code but increases review/rework is a regression.

2. Core metrics

Intent

• Signal precision/recall on labeled messages;
• duplicate/incorrect Work rate;
• acceptance-criteria correction rate;
• unresolved ambiguity discovered after implementation.

Design

• human design revision rate;
• implementation deviation rate;
• missed impacted components;
• slice replanning rate;
• architecture-review issues discovered after coding.

Execution

• terminal Work percentage;
• stale-running count;
• attempt success/retry/failure distribution;
• restart recovery success;
• duplicate side effects;
• cancellation cleanup;
• time/cost per verified slice.

Software quality

• verification pass rate on first candidate;
• escaped defect/regression rate;
• review changes requested;
• test weakening incidents;
• security/maintainability findings;
• integration-only failures;
• post-merge rollback/hotfix rate.

Human experience

• attention minutes per verified outcome;
• blocking questions per Work;
• time to understand review package;
• percentage of recommendations accepted;
• user override/correction rate.

Outcome

• achieved/partial/failed/inconclusive distribution;
• time from merge to result;
• original success signal movement;
• follow-up incident rate.

3. Evaluation corpus

Maintain a versioned fixture corpus.

Message/Signal fixtures

casual conversation
clear actionable request
multiple intents
duplicate request
new evidence for existing Work
conflicting requirement
incident/error Signal

Software Work fixtures

small isolated bug
API behavior addition
frontend/backend vertical flow
cross-package refactor
auth/permission change
database migration
parallel-slice conflict
verification repair
human question/resume
release regression

Each fixture includes:

repository/base SHA
input Signals
expected Work Definition properties
risk lane
expected impacted areas
required behavior/evidence
forbidden outcomes
human gold review notes

4. Evaluation modes

Deterministic unit/property evaluation

For:

• schemas;
• state transitions;
• dependency readiness;
• idempotency;
• budgets/retries;
• approval invalidation;
• actor recovery.

Must run on every change.

Adapter contract evaluation

Run the same suite against fake and live:

HarnessRuntime
SandboxRuntime
SourceControl
ArtifactStore
PreviewRuntime

Agent offline evaluation

Replay fixture inputs against candidate agent/Kit versions. Store all outputs and compare with current production baseline.

End-to-end sandbox evaluation

Run real repository tasks in isolated environments, verify exact candidate, and score review quality.

Dogfood/online evaluation

Observe real Work with human decisions, merge outcomes, incidents, and post-release results.

5. Scoring rubric

Score each candidate run 0–3 per dimension.

Score	Meaning
0	unsafe/incorrect/missing
1	substantial human repair required
2	acceptable with minor correction
3	correct, complete, reviewable

Dimensions:

intent fidelity
scope control
design correctness
slice quality
implementation correctness
test quality
maintainability
security/reliability
evidence completeness
review clarity
result achievement

Critical safety/invariant violation is automatic failure regardless of aggregate score.

6. Golden invariants

Automated assertions:

no duplicate Signal/Work/PR/deployment
no Work stuck active beyond lease/timeout
no verification without SHA/environment
no PR head different from verified SHA
no stale approval accepted
no cross-worktree mutation
no unscoped persistent credential
no builder-only completion verdict
no retry beyond policy
no silent canonical-knowledge mutation

7. Failure taxonomy

Classify failures consistently:

Intent
├── missed Signal
├── wrong Work
├── scope drift
└── unresolved ambiguity

Design
├── missed impact
├── wrong boundary
├── invalid slice
└── hidden trade-off

Execution
├── model failure
├── harness failure
├── sandbox failure
├── tool failure
├── timeout/cancellation
└── recovery/idempotency failure

Verification
├── missing check
├── flaky check
├── false pass
├── false failure
└── wrong revision/environment

Delivery
├── integration conflict
├── wrong PR revision
├── release failure
└── rollback failure

Outcome
├── user problem remains
├── regression
└── inconclusive observation

Every failed Work records one primary and optional contributing categories.

8. Experiment protocol

Any change to agent, prompt, model, Kit, tool, runtime policy, or Resolver heuristic requires:

hypothesis
target metric
candidate version
baseline version
fixture set
budget
results
regressions
decision

Recommended sequence:

offline corpus
→ shadow/dry run
→ small dogfood cohort
→ policy-controlled rollout

Do not promote based on one impressive example.

9. Agent/Kit promotion gate

A candidate may replace baseline only when:

• golden invariants pass;
• no critical regression;
• task success does not decrease materially;
• review/rework improves or remains acceptable;
• compute/time increase is justified;
• failure mode and rollback are known;
• version can be selected/reverted independently.

Record promotion as a durable decision.

10. Verifier evaluation

Verifiers themselves need evaluation.

Test:

• whether injected defects are caught;
• false-positive rate;
• whether checks bind exact candidate;
• whether design deviations are identified;
• whether evidence is concise and actionable;
• whether repair instructions reproduce the failure.

Use mutation/injected-defect suites where practical.

11. Human review calibration

Periodically compare:

agent/verifier verdict
human reviewer verdict
post-merge outcome

Track disagreement categories. A reviewer model agreeing with another model is not ground truth.

12. Post-release feedback

For completed Work, schedule/perform observation using defined success signals.

Convert:

• incidents;
• support complaints;
• failed health checks;
• rollback;
• unexpected metrics;
• manual rework

into linked Signals and evaluation records.

13. Improvement outputs

Evaluation may produce proposals for:

Work Definition rule
Design Packet template
slice heuristic
verification recipe
Kit/role/tool
runtime policy
repository knowledge
UI review package

Each proposal states evidence, expected benefit, risk, and rollback.

14. Dashboard minimum

Internal dogfood dashboard:

active/stale Work
terminal outcome distribution
first-pass verification
retry/replan causes
human wait/attention
duplicate side effects
review changes requested
post-merge failures
cost/time per verified Work
agent/Kit version comparison

Do not collapse quality into one vanity score.

15. Initial benchmark gate

Before v0 dogfood, pass:

✓ actionable vs casual Signal fixtures
✓ duplicate processing
✓ definition/design version invalidation
✓ fake Resolver crash recovery
✓ sandbox cancellation
✓ verification repair
✓ exact-SHA PR invariant
✓ human question/resume
✓ duplicate callback/PR prevention

Before increasing autonomy, add:

✓ integration conflict
✓ auth/permission fixture
✓ migration/rollback fixture
✓ post-release regression fixture

16. Continuous improvement loop

Run Work
→ capture structured outcome
→ classify failures
→ aggregate patterns
→ propose one bounded change
→ evaluate against corpus
→ dogfood
→ promote or revert

Improve the weakest stage, not the most fashionable model.