feat(kanban): durable multi-profile collaboration board (#17805)

Salvage of PR #16100 onto current main (after emozilla's #17514 fix
that unblocks plugin Pydantic body validation). History preserved on
the standing `feat/kanban-standing` branch; this squashes the 22
iterative commits into one clean landing.

What this lands:
- SQLite kernel (hermes_cli/kanban_db.py) — durable task board with
  tasks, task_links, task_runs, task_comments, task_events,
  kanban_notify_subs tables. WAL mode, atomic claim via CAS,
  tenant-namespaced, skills JSON array per task, max-runtime timeouts,
  worker heartbeats, idempotency keys, circuit breaker on repeated
  spawn failures, crash detection via /proc/<pid>/status, run history
  preserved across attempts.
- Dispatcher — runs inside the gateway by default
  (`kanban.dispatch_in_gateway: true`). Ticks every 60s, reclaims
  stale claims, promotes ready tasks, spawns `hermes -p <assignee>
  chat -q "work kanban task <id>"` with HERMES_KANBAN_TASK +
  HERMES_KANBAN_WORKSPACE env. Auto-loads `--skills kanban-worker`
  plus any per-task skills. Health telemetry warns on stuck ready
  queue.
- Structured tool surface (tools/kanban_tools.py) — 7 tools
  (kanban_show, kanban_complete, kanban_block, kanban_heartbeat,
  kanban_comment, kanban_create, kanban_link). Gated on
  HERMES_KANBAN_TASK via check_fn so zero schema footprint in normal
  sessions.
- System-prompt guidance (agent/prompt_builder.py KANBAN_GUIDANCE)
  injected only when kanban tools are active.
- Dashboard plugin (plugins/kanban/dashboard/) — Linear-style board
  UI: triage/todo/ready/running/blocked/done columns, drag-drop,
  inline create, task drawer with markdown, comments, run history,
  dependency editor, bulk ops, lanes-by-profile grouping, WS-driven
  live refresh. Matches active dashboard theme via CSS variables.
- CLI — `hermes kanban init|create|list|show|assign|link|unlink|
  claim|comment|complete|block|unblock|archive|tail|dispatch|context|
  init|gc|watch|stats|notify|log|heartbeat|runs|assignees` +
  `/kanban` slash in-session.
- Worker + orchestrator skills (skills/devops/kanban-worker +
  kanban-orchestrator) — pattern library for good summary/metadata
  shapes, retry diagnostics, block-reason examples, fan-out patterns.
- Per-task force-loaded skills — `--skill <name>` (repeatable),
  stored as JSON, threaded through to dispatcher argv as one
  `--skills X` pair per skill alongside the built-in kanban-worker.
  Dashboard + CLI + tool parity.
- Deprecation of standalone `hermes kanban daemon` — stub exits 2
  with migration guidance; `--force` escape hatch for headless hosts.
- Docs (website/docs/user-guide/features/kanban.md + kanban-tutorial.md)
  with 11 dashboard screenshots walking through four user stories
  (Solo Dev, Fleet Farming, Role Pipeline, Circuit Breaker).
- Tests (251 passing): kernel schema + migration + CAS atomicity,
  dispatcher logic, circuit breaker, crash detection, max-runtime
  timeouts, claim lifecycle, tenant isolation, idempotency keys, per-
  task skills round-trip + validation + dispatcher argv, tool surface
  (7 tools × round-trip + error paths), dashboard REST (CRUD + bulk
  + links + warnings), gateway-embedded dispatcher (config gate, env
  override, graceful shutdown), CLI deprecation stub, migration from
  legacy schemas.

Gateway integration:
- GatewayRunner._kanban_dispatcher_watcher — new asyncio background
  task, symmetric with _kanban_notifier_watcher. Runs dispatch_once
  via asyncio.to_thread so SQLite WAL never blocks the loop. Sleeps
  in 1s slices for snappy shutdown. Respects HERMES_KANBAN_DISPATCH_IN_GATEWAY=0
  env override for debugging.
- Config: new `kanban` section in DEFAULT_CONFIG with
  `dispatch_in_gateway: true` (default) + `dispatch_interval_seconds: 60`.
  Additive — no \_config_version bump needed.

Forward-compat:
- workflow_template_id / current_step_key columns on tasks (v1 writes
  NULL; v2 will use them for routing).
- task_runs holds claim machinery (claim_lock, claim_expires,
  worker_pid, last_heartbeat_at) so multi-attempt history is first-
  class from day one.

Closes #16102.

Co-authored-by: emozilla <emozilla@nousresearch.com>
This commit is contained in:
Teknium
2026-04-30 13:36:47 -07:00
committed by GitHub
parent 59c1a13f45
commit c868425467
48 changed files with 17420 additions and 0 deletions

View File

@@ -2732,6 +2732,17 @@ class GatewayRunner:
# Start background session expiry watcher to finalize expired sessions
asyncio.create_task(self._session_expiry_watcher())
# Start background kanban notifier — delivers `completed`, `blocked`,
# `spawn_auto_blocked`, and `crashed` events to gateway subscribers
# so human-in-the-loop workflows hear back without polling.
asyncio.create_task(self._kanban_notifier_watcher())
# Start background kanban dispatcher — spawns workers for ready
# tasks. Gated by `kanban.dispatch_in_gateway` (default True).
# When false, users run `hermes kanban daemon` externally or
# simply don't use kanban; this loop becomes a no-op.
asyncio.create_task(self._kanban_dispatcher_watcher())
# Start background reconnection watcher for platforms that failed at startup
if self._failed_platforms:
logger.info(
@@ -2907,6 +2918,399 @@ class GatewayRunner:
break
await asyncio.sleep(1)
async def _kanban_notifier_watcher(self, interval: float = 5.0) -> None:
"""Poll ``kanban_notify_subs`` and deliver terminal events to users.
For each subscription row, fetches ``task_events`` newer than the
stored cursor with kind in the terminal set (``completed``,
``blocked``, ``gave_up``, ``crashed``, ``timed_out``). Sends one
message per new event to ``(platform, chat_id, thread_id)``,
then advances the cursor. When a task reaches a terminal state
(``completed`` / ``archived``), the subscription is removed.
Runs in the gateway event loop; all SQLite work is pushed to a
thread via ``asyncio.to_thread`` so the loop never blocks on the
WAL lock. Failures in one tick don't stop subsequent ticks.
"""
from gateway.config import Platform as _Platform
try:
from hermes_cli import kanban_db as _kb
except Exception:
logger.warning("kanban notifier: kanban_db not importable; notifier disabled")
return
TERMINAL_KINDS = ("completed", "blocked", "gave_up", "crashed", "timed_out")
# Terminal event kinds trigger automatic unsubscription — the task
# is done, blocked, or in a retry-needed state that the human
# shouldn't keep pinging a stale chat for. Previously we only
# unsubbed when task.status in ('done', 'archived'), which left
# subscriptions on 'blocked' / 'gave_up' / 'crashed' / 'timed_out'
# tasks stranded forever.
TERMINAL_EVENT_KINDS = TERMINAL_KINDS
# Per-subscription send-failure counter. Adapter.send raising
# means the chat is dead (deleted, bot kicked, etc.) — after N
# consecutive send failures the sub is dropped so we don't spin
# against a dead chat every 5 seconds forever.
MAX_SEND_FAILURES = 3
sub_fail_counts: dict[tuple, int] = getattr(
self, "_kanban_sub_fail_counts", {}
)
self._kanban_sub_fail_counts = sub_fail_counts
# Initial delay so the gateway can finish wiring adapters.
await asyncio.sleep(5)
while self._running:
try:
def _collect():
conn = _kb.connect()
try:
_kb.init_db() # idempotent; handles first-run
except Exception:
pass
try:
subs = _kb.list_notify_subs(conn)
deliveries: list[dict] = []
for sub in subs:
cursor, events = _kb.unseen_events_for_sub(
conn,
task_id=sub["task_id"],
platform=sub["platform"],
chat_id=sub["chat_id"],
thread_id=sub.get("thread_id") or "",
kinds=TERMINAL_KINDS,
)
if not events:
continue
task = _kb.get_task(conn, sub["task_id"])
deliveries.append({
"sub": sub,
"cursor": cursor,
"events": events,
"task": task,
})
return deliveries
finally:
conn.close()
deliveries = await asyncio.to_thread(_collect)
for d in deliveries:
sub = d["sub"]
task = d["task"]
platform_str = (sub["platform"] or "").lower()
try:
plat = _Platform(platform_str)
except ValueError:
# Unknown platform string; skip and advance cursor so
# we don't replay forever.
await asyncio.to_thread(
self._kanban_advance, sub, d["cursor"],
)
continue
adapter = self.adapters.get(plat)
if adapter is None:
continue # platform not currently connected
title = (task.title if task else sub["task_id"])[:120]
for ev in d["events"]:
kind = ev.kind
# Identity prefix: attribute terminal pings to the
# worker that did the work. Makes fleets (where one
# chat subscribes to many tasks) legible at a glance.
who = (task.assignee if task and task.assignee else None)
tag = f"@{who} " if who else ""
if kind == "completed":
# Prefer the run's summary (the worker's
# intentional human-facing handoff, carried
# in the event payload), then fall back to
# task.result for legacy rows written before
# runs shipped.
handoff = ""
payload_summary = None
if ev.payload and ev.payload.get("summary"):
payload_summary = str(ev.payload["summary"])
if payload_summary:
h = payload_summary.strip().splitlines()[0][:200]
handoff = f"\n{h}"
elif task and task.result:
r = task.result.strip().splitlines()[0][:160]
handoff = f"\n{r}"
msg = (
f"{tag}Kanban {sub['task_id']} done"
f"{title}{handoff}"
)
elif kind == "blocked":
reason = ""
if ev.payload and ev.payload.get("reason"):
reason = f": {str(ev.payload['reason'])[:160]}"
msg = f"{tag}Kanban {sub['task_id']} blocked{reason}"
elif kind == "gave_up":
err = ""
if ev.payload and ev.payload.get("error"):
err = f"\n{str(ev.payload['error'])[:200]}"
msg = (
f"{tag}Kanban {sub['task_id']} gave up "
f"after repeated spawn failures{err}"
)
elif kind == "crashed":
msg = (
f"{tag}Kanban {sub['task_id']} worker crashed "
f"(pid gone); dispatcher will retry"
)
elif kind == "timed_out":
limit = 0
if ev.payload and ev.payload.get("limit_seconds"):
limit = int(ev.payload["limit_seconds"])
msg = (
f"{tag}Kanban {sub['task_id']} timed out "
f"(max_runtime={limit}s); will retry"
)
else:
continue
metadata: dict[str, Any] = {}
if sub.get("thread_id"):
metadata["thread_id"] = sub["thread_id"]
sub_key = (
sub["task_id"], sub["platform"],
sub["chat_id"], sub.get("thread_id") or "",
)
try:
await adapter.send(
sub["chat_id"], msg, metadata=metadata,
)
# Reset the failure counter on success.
sub_fail_counts.pop(sub_key, None)
except Exception as exc:
fails = sub_fail_counts.get(sub_key, 0) + 1
sub_fail_counts[sub_key] = fails
logger.warning(
"kanban notifier: send failed for %s on %s "
"(attempt %d/%d): %s",
sub["task_id"], platform_str, fails,
MAX_SEND_FAILURES, exc,
)
if fails >= MAX_SEND_FAILURES:
logger.warning(
"kanban notifier: dropping subscription "
"%s on %s after %d consecutive send failures",
sub["task_id"], platform_str, fails,
)
await asyncio.to_thread(self._kanban_unsub, sub)
sub_fail_counts.pop(sub_key, None)
# Don't advance cursor on send failure — retry next tick.
break
else:
# All events delivered; advance cursor + maybe unsub.
await asyncio.to_thread(
self._kanban_advance, sub, d["cursor"],
)
# Unsubscribe when the LAST delivered event is a
# terminal kind (the task hit a "no further updates"
# state), not just on task.status in {done, archived}.
# Covers blocked / gave_up / crashed / timed_out which
# used to leak subs forever.
last_kind = d["events"][-1].kind if d["events"] else None
task_terminal = task and task.status in ("done", "archived")
event_terminal = last_kind in TERMINAL_EVENT_KINDS
if task_terminal or event_terminal:
await asyncio.to_thread(
self._kanban_unsub, sub,
)
except Exception as exc:
logger.warning("kanban notifier tick failed: %s", exc)
# Sleep with cancellation checks.
for _ in range(int(max(1, interval))):
if not self._running:
return
await asyncio.sleep(1)
def _kanban_advance(self, sub: dict, cursor: int) -> None:
"""Sync helper: advance a subscription's cursor. Runs in to_thread."""
from hermes_cli import kanban_db as _kb
conn = _kb.connect()
try:
_kb.advance_notify_cursor(
conn,
task_id=sub["task_id"],
platform=sub["platform"],
chat_id=sub["chat_id"],
thread_id=sub.get("thread_id") or "",
new_cursor=cursor,
)
finally:
conn.close()
def _kanban_unsub(self, sub: dict) -> None:
from hermes_cli import kanban_db as _kb
conn = _kb.connect()
try:
_kb.remove_notify_sub(
conn,
task_id=sub["task_id"],
platform=sub["platform"],
chat_id=sub["chat_id"],
thread_id=sub.get("thread_id") or "",
)
finally:
conn.close()
async def _kanban_dispatcher_watcher(self) -> None:
"""Embedded kanban dispatcher — one tick every `dispatch_interval_seconds`.
Gated by `kanban.dispatch_in_gateway` in config.yaml (default True).
When true, the gateway hosts the single dispatcher for this profile:
no separate `hermes kanban daemon` process needed. When false, the
loop exits immediately and an external daemon is expected.
Each tick calls :func:`kanban_db.dispatch_once` inside
``asyncio.to_thread`` so the SQLite WAL lock never blocks the
event loop. Failures in one tick don't stop subsequent ticks —
same pattern as `_kanban_notifier_watcher`.
Shutdown: the loop checks ``self._running`` between ticks; gateway
stop() flips it to False and cancels pending tasks, and the
in-flight ``to_thread`` returns on its own after the current
``dispatch_once`` call finishes (typically <1ms on an idle board).
"""
# Read config once at boot. If the user flips the flag later, they
# restart the gateway; same pattern as every other background
# watcher here. Honours HERMES_KANBAN_DISPATCH_IN_GATEWAY env var
# as an escape hatch (false-y value disables without editing YAML).
try:
from hermes_cli.config import load_config as _load_config
except Exception:
logger.warning("kanban dispatcher: config loader unavailable; disabled")
return
env_override = os.environ.get("HERMES_KANBAN_DISPATCH_IN_GATEWAY", "").strip().lower()
if env_override in ("0", "false", "no", "off"):
logger.info("kanban dispatcher: disabled via HERMES_KANBAN_DISPATCH_IN_GATEWAY env")
return
try:
cfg = _load_config()
except Exception as exc:
logger.warning("kanban dispatcher: cannot load config (%s); disabled", exc)
return
kanban_cfg = cfg.get("kanban", {}) if isinstance(cfg, dict) else {}
if not kanban_cfg.get("dispatch_in_gateway", True):
logger.info(
"kanban dispatcher: disabled via config kanban.dispatch_in_gateway=false"
)
return
try:
from hermes_cli import kanban_db as _kb
except Exception:
logger.warning("kanban dispatcher: kanban_db not importable; dispatcher disabled")
return
interval = float(kanban_cfg.get("dispatch_interval_seconds", 60) or 60)
if interval < 1.0:
interval = 1.0 # sanity floor — tighter than this is a footgun
# Initial delay so the gateway finishes wiring adapters before the
# dispatcher spawns workers (those workers may hit gateway notify
# subscriptions etc.). Matches the notifier watcher's delay.
await asyncio.sleep(5)
# Health telemetry mirrored from `_cmd_daemon`: warn when ready
# queue is non-empty but spawns are 0 for N consecutive ticks —
# usually means broken PATH, missing venv, or credential loss.
HEALTH_WINDOW = 6
bad_ticks = 0
last_warn_at = 0
def _tick_once() -> "Optional[object]":
"""Run one dispatch_once; return result or None on error.
Runs in a worker thread via `asyncio.to_thread`."""
conn = None
try:
conn = _kb.connect()
try:
_kb.init_db() # idempotent, handles first-run
except Exception:
pass
return _kb.dispatch_once(conn)
except Exception:
logger.exception("kanban dispatcher: tick failed")
return None
finally:
if conn is not None:
try:
conn.close()
except Exception:
pass
def _ready_nonempty() -> bool:
"""Cheap probe: is there at least one ready+assigned+unclaimed task?"""
conn = None
try:
conn = _kb.connect()
row = conn.execute(
"SELECT 1 FROM tasks "
"WHERE status = 'ready' AND assignee IS NOT NULL "
" AND claim_lock IS NULL LIMIT 1"
).fetchone()
return row is not None
except Exception:
return False
finally:
if conn is not None:
try:
conn.close()
except Exception:
pass
logger.info(
"kanban dispatcher: embedded in gateway (interval=%.1fs)", interval
)
while self._running:
try:
res = await asyncio.to_thread(_tick_once)
if res is not None and getattr(res, "spawned", None):
# Quiet by default — only log when something actually
# happened, so an idle gateway stays silent.
logger.info(
"kanban dispatcher: tick spawned=%d reclaimed=%d "
"crashed=%d timed_out=%d promoted=%d auto_blocked=%d",
len(res.spawned),
res.reclaimed,
len(res.crashed) if hasattr(res.crashed, "__len__") else 0,
len(res.timed_out) if hasattr(res.timed_out, "__len__") else 0,
res.promoted,
len(res.auto_blocked) if hasattr(res.auto_blocked, "__len__") else 0,
)
# Health telemetry
ready_pending = await asyncio.to_thread(_ready_nonempty)
spawned_any = bool(res and getattr(res, "spawned", None))
if ready_pending and not spawned_any:
bad_ticks += 1
else:
bad_ticks = 0
if bad_ticks >= HEALTH_WINDOW:
now = int(time.time())
if now - last_warn_at >= 300:
logger.warning(
"kanban dispatcher stuck: ready queue non-empty for "
"%d consecutive ticks but 0 workers spawned. Check "
"profile health (venv, PATH, credentials) and "
"`hermes kanban list --status ready`.",
bad_ticks,
)
last_warn_at = now
except asyncio.CancelledError:
logger.debug("kanban dispatcher: cancelled")
raise
except Exception:
logger.exception("kanban dispatcher: unexpected watcher error")
# Sleep in 1s slices so shutdown is snappy — otherwise a stop()
# waits up to `interval` seconds for the current sleep to finish.
slept = 0.0
while slept < interval and self._running:
await asyncio.sleep(min(1.0, interval - slept))
slept += 1.0
async def _platform_reconnect_watcher(self) -> None:
"""Background task that periodically retries connecting failed platforms.
@@ -4168,6 +4572,14 @@ class GatewayRunner:
if _cmd_def_inner and _cmd_def_inner.name == "background":
return await self._handle_background_command(event)
# /kanban must bypass the guard. It writes to a profile-agnostic
# DB (kanban.db), not to the running agent's state. In fact
# /kanban unblock is often the only way to free a worker that
# has blocked waiting for a peer — letting that be dispatched
# mid-run is the whole point of the board.
if _cmd_def_inner and _cmd_def_inner.name == "kanban":
return await self._handle_kanban_command(event)
# Session-level toggles that are safe to run mid-agent —
# /yolo can unblock a pending approval prompt, /verbose cycles
# the tool-progress display mode for the ongoing stream.
@@ -4415,6 +4827,9 @@ class GatewayRunner:
if canonical == "personality":
return await self._handle_personality_command(event)
if canonical == "kanban":
return await self._handle_kanban_command(event)
if canonical == "retry":
return await self._handle_retry_command(event)
@@ -6031,6 +6446,84 @@ class GatewayRunner:
return "\n".join(lines)
async def _handle_kanban_command(self, event: MessageEvent) -> str:
"""Handle /kanban — delegate to the shared kanban CLI.
Run the potentially-blocking DB work in a thread pool so the
gateway event loop stays responsive. Read operations (list,
show, context, tail) are permitted while an agent is running;
mutations are allowed too because the board is profile-agnostic
and does not touch the running agent's state.
For ``/kanban create`` invocations we also auto-subscribe the
originating gateway source (platform + chat + thread) to the new
task's terminal events, so the user hears back when the worker
completes / blocks / auto-blocks / crashes without having to poll.
"""
import asyncio
import re
from hermes_cli.kanban import run_slash
text = (event.text or "").strip()
# Strip the leading "/kanban" (with or without slash), leaving args.
if text.startswith("/"):
text = text.lstrip("/")
if text.startswith("kanban"):
text = text[len("kanban"):].lstrip()
is_create = text.split(None, 1)[:1] == ["create"]
try:
output = await asyncio.to_thread(run_slash, text)
except Exception as exc: # pragma: no cover - defensive
return f"⚠ kanban error: {exc}"
# Auto-subscribe on create. Parse the task id from the CLI's standard
# success line ("Created t_abcd (ready, assignee=...)"). If the user
# passed --json we don't subscribe; they're clearly scripting and
# can call /kanban notify-subscribe explicitly.
if is_create and output:
m = re.search(r"Created\s+(t_[0-9a-f]+)\b", output)
if m:
task_id = m.group(1)
try:
source = event.source
platform = getattr(source, "platform", None)
platform_str = (
platform.value if hasattr(platform, "value") else str(platform or "")
).lower()
chat_id = str(getattr(source, "chat_id", "") or "")
thread_id = str(getattr(source, "thread_id", "") or "")
user_id = str(getattr(source, "user_id", "") or "") or None
if platform_str and chat_id:
def _sub():
from hermes_cli import kanban_db as _kb
conn = _kb.connect()
try:
_kb.add_notify_sub(
conn, task_id=task_id,
platform=platform_str, chat_id=chat_id,
thread_id=thread_id or None,
user_id=user_id,
)
finally:
conn.close()
await asyncio.to_thread(_sub)
output = (
output.rstrip()
+ f"\n(subscribed — you'll be notified when {task_id} "
f"completes or blocks)"
)
except Exception as exc:
logger.warning("kanban create auto-subscribe failed: %s", exc)
# Gateway messages have practical length caps; truncate long
# listings to keep the UX reasonable.
if len(output) > 3800:
output = output[:3800] + "\n… (truncated; use `hermes kanban …` in your terminal for full output)"
return output or "(no output)"
async def _handle_status_command(self, event: MessageEvent) -> str:
"""Handle /status command."""
source = event.source