feat: Slice 1 polish - MiMo V2.5 model, visible reasoning, image attachments, mobile keyboard fix
- Switch conversation agent to xiaomi/mimo-v2.5 (multimodal: text + image) - Render native reasoning parts as live 'Thinking trace' (streaming open, collapsed after completion); inline <think> extraction for streaming models - Image attachments: picker (up to 4, 10MB each), base64 to Flue AgentPromptImage, authenticated blob-URL replay for historical images - Mobile keyboard viewport fix: visual-viewport hook, fixed shell, interactive-widget=resizes-content, header pinned, composer follows keyboard - Conversation to Signal to proposed Work: Convex persistence, Effect validation in @code/work-os, Work cards with exact source provenance - Streamdown markdown + Mermaid chart rendering in chat messages - Flue tool turns hidden, reasoning-containing turns remain visible - Frontend regression tests: keyboard viewport, responsive shell, attachment overflow, authenticated images, reasoning traces, transforms - .env.example updated to xiaomi/mimo-v2.5 config
This commit is contained in:
-Puter
405
docs/evaluation.md
Normal file
405
docs/evaluation.md
Normal file
@@ -0,0 +1,405 @@
|
||||
# Zopu Work OS — Evaluation and Improvement
|
||||
|
||||
> **Purpose:** measure whether Zopu produces better software and improve agents, Kits, tools, prompts, and workflow without regressions.
|
||||
> **Principle:** no autonomy increase without evidence.
|
||||
|
||||
## 1. Evaluation targets
|
||||
|
||||
Evaluate five layers separately:
|
||||
|
||||
```text
|
||||
A. Intent quality
|
||||
B. Design quality
|
||||
C. Execution reliability
|
||||
D. Software quality
|
||||
E. Product outcome
|
||||
```
|
||||
|
||||
A fast agent that generates more code but increases review/rework is a regression.
|
||||
|
||||
## 2. Core metrics
|
||||
|
||||
### Intent
|
||||
|
||||
- Signal precision/recall on labeled messages;
|
||||
- duplicate/incorrect Work rate;
|
||||
- acceptance-criteria correction rate;
|
||||
- unresolved ambiguity discovered after implementation.
|
||||
|
||||
### Design
|
||||
|
||||
- human design revision rate;
|
||||
- implementation deviation rate;
|
||||
- missed impacted components;
|
||||
- slice replanning rate;
|
||||
- architecture-review issues discovered after coding.
|
||||
|
||||
### Execution
|
||||
|
||||
- terminal Work percentage;
|
||||
- stale-running count;
|
||||
- attempt success/retry/failure distribution;
|
||||
- restart recovery success;
|
||||
- duplicate side effects;
|
||||
- cancellation cleanup;
|
||||
- time/cost per verified slice.
|
||||
|
||||
### Software quality
|
||||
|
||||
- verification pass rate on first candidate;
|
||||
- escaped defect/regression rate;
|
||||
- review changes requested;
|
||||
- test weakening incidents;
|
||||
- security/maintainability findings;
|
||||
- integration-only failures;
|
||||
- post-merge rollback/hotfix rate.
|
||||
|
||||
### Human experience
|
||||
|
||||
- attention minutes per verified outcome;
|
||||
- blocking questions per Work;
|
||||
- time to understand review package;
|
||||
- percentage of recommendations accepted;
|
||||
- user override/correction rate.
|
||||
|
||||
### Outcome
|
||||
|
||||
- achieved/partial/failed/inconclusive distribution;
|
||||
- time from merge to result;
|
||||
- original success signal movement;
|
||||
- follow-up incident rate.
|
||||
|
||||
## 3. Evaluation corpus
|
||||
|
||||
Maintain a versioned fixture corpus.
|
||||
|
||||
### Message/Signal fixtures
|
||||
|
||||
```text
|
||||
casual conversation
|
||||
clear actionable request
|
||||
multiple intents
|
||||
duplicate request
|
||||
new evidence for existing Work
|
||||
conflicting requirement
|
||||
incident/error Signal
|
||||
```
|
||||
|
||||
### Software Work fixtures
|
||||
|
||||
```text
|
||||
small isolated bug
|
||||
API behavior addition
|
||||
frontend/backend vertical flow
|
||||
cross-package refactor
|
||||
auth/permission change
|
||||
database migration
|
||||
parallel-slice conflict
|
||||
verification repair
|
||||
human question/resume
|
||||
release regression
|
||||
```
|
||||
|
||||
Each fixture includes:
|
||||
|
||||
```text
|
||||
repository/base SHA
|
||||
input Signals
|
||||
expected Work Definition properties
|
||||
risk lane
|
||||
expected impacted areas
|
||||
required behavior/evidence
|
||||
forbidden outcomes
|
||||
human gold review notes
|
||||
```
|
||||
|
||||
## 4. Evaluation modes
|
||||
|
||||
### Deterministic unit/property evaluation
|
||||
|
||||
For:
|
||||
|
||||
- schemas;
|
||||
- state transitions;
|
||||
- dependency readiness;
|
||||
- idempotency;
|
||||
- budgets/retries;
|
||||
- approval invalidation;
|
||||
- actor recovery.
|
||||
|
||||
Must run on every change.
|
||||
|
||||
### Adapter contract evaluation
|
||||
|
||||
Run the same suite against fake and live:
|
||||
|
||||
```text
|
||||
HarnessRuntime
|
||||
SandboxRuntime
|
||||
SourceControl
|
||||
ArtifactStore
|
||||
PreviewRuntime
|
||||
```
|
||||
|
||||
### Agent offline evaluation
|
||||
|
||||
Replay fixture inputs against candidate agent/Kit versions. Store all outputs and compare with current production baseline.
|
||||
|
||||
### End-to-end sandbox evaluation
|
||||
|
||||
Run real repository tasks in isolated environments, verify exact candidate, and score review quality.
|
||||
|
||||
### Dogfood/online evaluation
|
||||
|
||||
Observe real Work with human decisions, merge outcomes, incidents, and post-release results.
|
||||
|
||||
## 5. Scoring rubric
|
||||
|
||||
Score each candidate run 0–3 per dimension.
|
||||
|
||||
| Score | Meaning |
|
||||
|---|---|
|
||||
| 0 | unsafe/incorrect/missing |
|
||||
| 1 | substantial human repair required |
|
||||
| 2 | acceptable with minor correction |
|
||||
| 3 | correct, complete, reviewable |
|
||||
|
||||
Dimensions:
|
||||
|
||||
```text
|
||||
intent fidelity
|
||||
scope control
|
||||
design correctness
|
||||
slice quality
|
||||
implementation correctness
|
||||
test quality
|
||||
maintainability
|
||||
security/reliability
|
||||
evidence completeness
|
||||
review clarity
|
||||
result achievement
|
||||
```
|
||||
|
||||
Critical safety/invariant violation is automatic failure regardless of aggregate score.
|
||||
|
||||
## 6. Golden invariants
|
||||
|
||||
Automated assertions:
|
||||
|
||||
```text
|
||||
no duplicate Signal/Work/PR/deployment
|
||||
no Work stuck active beyond lease/timeout
|
||||
no verification without SHA/environment
|
||||
no PR head different from verified SHA
|
||||
no stale approval accepted
|
||||
no cross-worktree mutation
|
||||
no unscoped persistent credential
|
||||
no builder-only completion verdict
|
||||
no retry beyond policy
|
||||
no silent canonical-knowledge mutation
|
||||
```
|
||||
|
||||
## 7. Failure taxonomy
|
||||
|
||||
Classify failures consistently:
|
||||
|
||||
```text
|
||||
Intent
|
||||
├── missed Signal
|
||||
├── wrong Work
|
||||
├── scope drift
|
||||
└── unresolved ambiguity
|
||||
|
||||
Design
|
||||
├── missed impact
|
||||
├── wrong boundary
|
||||
├── invalid slice
|
||||
└── hidden trade-off
|
||||
|
||||
Execution
|
||||
├── model failure
|
||||
├── harness failure
|
||||
├── sandbox failure
|
||||
├── tool failure
|
||||
├── timeout/cancellation
|
||||
└── recovery/idempotency failure
|
||||
|
||||
Verification
|
||||
├── missing check
|
||||
├── flaky check
|
||||
├── false pass
|
||||
├── false failure
|
||||
└── wrong revision/environment
|
||||
|
||||
Delivery
|
||||
├── integration conflict
|
||||
├── wrong PR revision
|
||||
├── release failure
|
||||
└── rollback failure
|
||||
|
||||
Outcome
|
||||
├── user problem remains
|
||||
├── regression
|
||||
└── inconclusive observation
|
||||
```
|
||||
|
||||
Every failed Work records one primary and optional contributing categories.
|
||||
|
||||
## 8. Experiment protocol
|
||||
|
||||
Any change to agent, prompt, model, Kit, tool, runtime policy, or Resolver heuristic requires:
|
||||
|
||||
```text
|
||||
hypothesis
|
||||
target metric
|
||||
candidate version
|
||||
baseline version
|
||||
fixture set
|
||||
budget
|
||||
results
|
||||
regressions
|
||||
decision
|
||||
```
|
||||
|
||||
Recommended sequence:
|
||||
|
||||
```text
|
||||
offline corpus
|
||||
→ shadow/dry run
|
||||
→ small dogfood cohort
|
||||
→ policy-controlled rollout
|
||||
```
|
||||
|
||||
Do not promote based on one impressive example.
|
||||
|
||||
## 9. Agent/Kit promotion gate
|
||||
|
||||
A candidate may replace baseline only when:
|
||||
|
||||
- golden invariants pass;
|
||||
- no critical regression;
|
||||
- task success does not decrease materially;
|
||||
- review/rework improves or remains acceptable;
|
||||
- compute/time increase is justified;
|
||||
- failure mode and rollback are known;
|
||||
- version can be selected/reverted independently.
|
||||
|
||||
Record promotion as a durable decision.
|
||||
|
||||
## 10. Verifier evaluation
|
||||
|
||||
Verifiers themselves need evaluation.
|
||||
|
||||
Test:
|
||||
|
||||
- whether injected defects are caught;
|
||||
- false-positive rate;
|
||||
- whether checks bind exact candidate;
|
||||
- whether design deviations are identified;
|
||||
- whether evidence is concise and actionable;
|
||||
- whether repair instructions reproduce the failure.
|
||||
|
||||
Use mutation/injected-defect suites where practical.
|
||||
|
||||
## 11. Human review calibration
|
||||
|
||||
Periodically compare:
|
||||
|
||||
```text
|
||||
agent/verifier verdict
|
||||
human reviewer verdict
|
||||
post-merge outcome
|
||||
```
|
||||
|
||||
Track disagreement categories. A reviewer model agreeing with another model is not ground truth.
|
||||
|
||||
## 12. Post-release feedback
|
||||
|
||||
For completed Work, schedule/perform observation using defined success signals.
|
||||
|
||||
Convert:
|
||||
|
||||
- incidents;
|
||||
- support complaints;
|
||||
- failed health checks;
|
||||
- rollback;
|
||||
- unexpected metrics;
|
||||
- manual rework
|
||||
|
||||
into linked Signals and evaluation records.
|
||||
|
||||
## 13. Improvement outputs
|
||||
|
||||
Evaluation may produce proposals for:
|
||||
|
||||
```text
|
||||
Work Definition rule
|
||||
Design Packet template
|
||||
slice heuristic
|
||||
verification recipe
|
||||
Kit/role/tool
|
||||
runtime policy
|
||||
repository knowledge
|
||||
UI review package
|
||||
```
|
||||
|
||||
Each proposal states evidence, expected benefit, risk, and rollback.
|
||||
|
||||
## 14. Dashboard minimum
|
||||
|
||||
Internal dogfood dashboard:
|
||||
|
||||
```text
|
||||
active/stale Work
|
||||
terminal outcome distribution
|
||||
first-pass verification
|
||||
retry/replan causes
|
||||
human wait/attention
|
||||
duplicate side effects
|
||||
review changes requested
|
||||
post-merge failures
|
||||
cost/time per verified Work
|
||||
agent/Kit version comparison
|
||||
```
|
||||
|
||||
Do not collapse quality into one vanity score.
|
||||
|
||||
## 15. Initial benchmark gate
|
||||
|
||||
Before v0 dogfood, pass:
|
||||
|
||||
```text
|
||||
✓ actionable vs casual Signal fixtures
|
||||
✓ duplicate processing
|
||||
✓ definition/design version invalidation
|
||||
✓ fake Resolver crash recovery
|
||||
✓ sandbox cancellation
|
||||
✓ verification repair
|
||||
✓ exact-SHA PR invariant
|
||||
✓ human question/resume
|
||||
✓ duplicate callback/PR prevention
|
||||
```
|
||||
|
||||
Before increasing autonomy, add:
|
||||
|
||||
```text
|
||||
✓ integration conflict
|
||||
✓ auth/permission fixture
|
||||
✓ migration/rollback fixture
|
||||
✓ post-release regression fixture
|
||||
```
|
||||
|
||||
## 16. Continuous improvement loop
|
||||
|
||||
```text
|
||||
Run Work
|
||||
→ capture structured outcome
|
||||
→ classify failures
|
||||
→ aggregate patterns
|
||||
→ propose one bounded change
|
||||
→ evaluate against corpus
|
||||
→ dogfood
|
||||
→ promote or revert
|
||||
```
|
||||
|
||||
Improve the weakest stage, not the most fashionable model.
|
||||
Reference in New Issue
Block a user