Gavel: Durable Multi-Agent Pipeline Engineering

Gavel is a local idea validation workbench. You keep a versioned worksheet, log evidence, and run a five-role panel (VC, engineer, PM, customer, competitor) that debates, re-votes, and returns a structured GO / ITERATE / NO-GO. This write-up is about the systems underneath that surface.

I spent more time on failure modes than on prompts. Browser refresh killing a run. A 9B model returning five copies of the same concern. A typo edit rewriting the worksheet under an old score. Those bugs look boring until they show up in a live session, and most of the interesting code is there because of them.

Product loop and design choices are in the companion blog post. This page is architecture, contracts, and testing.

I treat production orchestration as a workflow. Application code owns readiness, speaker order, retries, event sequence, and completion. Models fill judgment fields inside Pydantic schemas. I tried agent-led orchestration early; it looked clean when the model cooperated and fell apart on local hardware.

• POST /api/runs returns a run_id immediately. RunManager.ensure_started() kicks stream_pipeline() exactly once.
• pipeline.py is two phases: ThreadPoolExecutor(max_workers=5) for the roast panel, then a compiled LangGraph for debate.
• Every pipeline event lands in runs.db with a monotonic sequence. SSE clients replay, then wait on in-process wakeups.
• The Next.js client folds envelopes through a pure TypeScript reducer (~580 lines, 19 event types) so live and reconnect paths converge.
• verification/ is shared by runtime guardrails, eval scorers, and UI degradation hints. Same Jaccard thresholds, same herding checks.

A clean path is roughly fifteen model calls before you count research or retries. Readiness is pure Python. Everything after that is gated and instrumented.

Debate routing is boring on purpose. build_debate_graph() adds five speaker nodes in fixed order (vc → engineer → pm → customer → competitor), an advance_round node, optional revote, and moderator. Conditional edges after each speaker call route_next_speaker; the model never picks the next node. Entry is always vc. After the last speaker in a round: loop while round < max_debate_rounds (default 3), else revote → moderator → END.

Re-vote exists because a transcript without score movement is theater. Each judge re-scores against the full debate under a ±3 cap, and a score change has to update its reason. assess_revote_quality flags herd deltas (≥4 judges moving by the same amount) and unexplained moves. Those checks live in verification/panel.py and run in both production and evals.

Abort checks are threaded into speaker and revote nodes so cooperative cancel can stop between turns without tearing down the LangGraph mid-token.

Streaming was the first thing that looked finished and was not. If the pipeline dies with the HTTP connection, you have a demo. RunManager + SQLite split writers from readers:

GET /api/runs/{id}/events is a subscriber, not the execution owner. The handler parses Last-Event-ID, calls manager.subscribe(run_id, after_sequence=…), and emits heartbeats on idle so proxies do not close the stream:

id: 42
data: {"type":"judge_verdict_completed","run_id":"…","sequence":42,"payload":{…},"created_at":"…"}

: heartbeat

id: 43
data: {"type":"debate_token_delta","run_id":"…","sequence":43,"payload":{"speaker":"vc","round":1,"delta":"…"},"created_at":"…"}

Perceived streaming, not raw provider-to-browser forwarding. The panel finishes all five calls (including retries) before verdict events emit in display order. Debate tokens reach SSE after a speaker turn is collected. Progress and replay are real; I did not build token forwarding through the whole stack.

Single-process assumption is intentional for v0.8. In-memory task handles and subscriber wakeups are simple with one uvicorn worker. Two workers would need a shared queue and pub/sub. The event log is already durable; the coordination layer is not.

Rate limits sit on the public write paths:

Internal pipeline events are frozen dataclasses in src/events.py. The API maps them to a frontend-safe envelope before they hit SQLite or SSE. Naming is mechanical: JudgeVerdictCompleted becomes judge_verdict_completed, with two special cases for completion and metrics.

# src/api/events.py — dataclass → snake_case SSE type
def pipeline_event_type(event: PipelineEvent) -> str:
    if isinstance(event, PipelineCompleted):
        return "run_completed"
    if isinstance(event, RunMetrics):
        return "run_metrics"
    return _camel_to_snake(type(event).__name__)
    # JudgeVerdictCompleted → judge_verdict_completed

// Every SSE data frame is the same shape
type ApiEventEnvelope = {
  type: string
  run_id: string
  sequence: number   // monotonic; Last-Event-ID on reconnect
  payload: Record<string, unknown>
  created_at: string
}

Nineteen event types the reducer handles. Typical happy-path order for a full roast:

A few payload details that matter for UI correctness:

• judge_verdict_completed carries the full Verdict plus completed/total so the panel can animate progress without waiting for roast_panel_completed.
• debate_token_delta only has speaker, round, and delta. The reducer upserts a DebateTurnView and concatenates; debate_message_published freezes the turn.
• revote_started snapshots current scores into revoteBaseline so the UI can show deltas after revote_judge_completed.
• revote_judge_completed may include change_reason only when the score actually moved (taken from evidence_to_change_verdict).
• run_completed attaches panel_quality from the shared verification module (lens uniqueness, generic rate, warnings) so the UI and evals agree.
• appeal_completed is not part of the pipeline stream during the roast; it is appended later when POST /appeal finishes, and open SSE subscribers still receive it.

The workbench does not own run state. It reduces ordered envelopes. That sounds obvious until you debug a reconnect that double-applies a re-vote or drops a debate token delta.

// web/src/lib/sse/run-reducer.ts
export function runReducer(state: RunState, envelope: ApiEventEnvelope): RunState {
  // Reconnect / multi-tab safety: ignore gaps already applied
  if (envelope.sequence <= state.lastSequence) return state

  switch (envelope.type) {
    case "debate_token_delta":
      // merge into DebateTurnView.content; keep streaming=true
      …
    case "revote_judge_completed":
      // update judge verdict; store change_reason for delta badges
      …
    case "run_completed":
      // attach panel_quality from payload; status=completed
      …
  }
}

• Sequence gate first: envelope.sequence ≤ lastSequence → no-op. Live and Last-Event-ID replay converge to the same RunState.
• Judge order is a shared const (vc, engineer, pm, customer, competitor) on both sides so role identity cannot drift.
• judges_dispatched flips all five JudgeView statuses to thinking before any verdict arrives.
• On run_failed mid-panel, pending judges (idle/thinking) are marked failed so the UI does not spin forever.
• Observability lands as run_metrics: roast/debate/revote seconds, tokens, estimated_cost_usd, model_runtime local|deepseek, per-call breakdowns.
• Cost uses fixed DeepSeek rates ($0.14/M in, $0.28/M out) or $0 for local; token counts fall back to chars÷4 when providers omit usage.

Schema validation catches missing fields. It does not catch five judges saying the same thing with different wording. The Verdict model is the contract between the LLM, the event log, the handoff builder, and the UI:

Panel quality sits on top of Pydantic. Weak panels retry ≤2 times with role-specific nudges. Uniform panels can finish with a low-confidence flag. Lens collapse (Jaccard ≥ 0.85 on concerns/evidence, or >40% generic evidence phrases) fails the run.

Synthesis is its own schema: overall_recommendation (GO / ITERATE / NO-GO), confidence, top strengths/risks/problems (max 3 each), biggest_disagreement, recommended_experiment. That structured object ships on debate_completed and is what handoff ingest reads, not free-form chat.

Three SQLite files: runs.db (event log), workspaces.db (versions + ledger), ideas.db (compact memory, optional 768-dim vectors). Memory prompts get summaries only, never full old transcripts. Cross-store consistency is the ongoing tax of that split.

Twelve golden ideas sit under three tiers. Tier 0 is free structural CI on every PR. Tier 1 requires pass_rate = 1.0 on the golden set (full panels, debate message counts, lens differentiation, re-vote quality, appeal discrimination). Tier 2 is a monthly DeepSeek grader (~$0.50–2) that flags any dimension drop ≥ 0.5. Keyword matching is not the pass signal on purpose.

Test surface around the pipeline:

CI runs Ruff, Python 3.11–3.13, structural eval fixtures, web unit tests, Playwright, and a Docker build.

• Fixed graphs beat agent planners for this workload. A missing judge is a bug.
• Decoupling HTTP from the pipeline turned refresh and multi-tab watching into ordinary subscriber behavior.
• Warn on bland consensus, fail on collapsed lenses. Mixing those policies either blocks too much or ships fake specialization.
• Sharing verification between runtime and evals kept prompt changes honest. When the scorer and the gate disagree, one of them is wrong.
• Copy-on-write worksheets only help if the API refuses to mutate locked inputs. The UI benefit is a side effect of that store rule.
• Three SQLite files and one worker were fine early. Cross-store joins and multi-worker coordination are what you pay later.

• Single uvicorn worker: background tasks and in-process SSE wakeups are not coordinated across workers.
• No multi-user auth. Fine for self-host; a public deploy needs authz on every run and export path.
• Appeal re-scores selected judges only. No second debate. One appeal per run.
• Cancel and wall-clock budget are cooperative. In-flight judge calls may finish after cancel.
• DeepAgents path exists for experiments but is Streamlit-only. API and Next.js stay on the deterministic pipeline.
• Cost estimates are static rates plus a chars÷4 fallback. Good enough for orientation, not billing.