Building Gavel: Turning AI Opinions Into Experiments

The product boundary

Gavel is organized around a versioned idea worksheet: audience, problem, current workaround, proposed solution, pricing hypothesis, competitors, and the riskiest assumption. Around that document live assumptions, evidence, experiments, and interview notes. A five-role judge panel can evaluate a version once the idea is specific enough to evaluate.

The first prototype stopped at the panel. Five roles scored a pitch, argued, synthesized, and the founder left with a label. That is a demo. It is not a habit. The useful product question is uglier: which assumption is weakest, what evidence would change the answer, and what can someone do this week? Once I took that seriously, the panel became one station on a longer loop.

Almost everything hard in the codebase exists because of that loop: readiness policy, durable runs, copy-on-write versions, panel quality checks, handoffs, appeals, research, and assist endpoints. The flashy part is judgment. The engineering is keeping judgment from lying about its input or evaporating into chat.

Own the control plane

I tried agent-led orchestration early. It looked elegant when the model cooperated. Smaller local models did not reliably call all five judges. A missing engineer or customer is not a creative variation. It is a broken run.

The production path is boring on purpose. Five initial verdicts run concurrently. A fixed LangGraph state machine controls debate order and round count. After the transcript, every judge re-votes in parallel under a score-move cap. The moderator emits a structured synthesis. Models write the substance. Application code decides whether a role is skipped, whether another round appears, and whether malformed output is accepted.

That split shows up in call sites, not in a slide titled "guardrails." If a policy matters, it lives next to unit tests. If a field is judgment, it lives in a schema the model fills.

Readiness is policy, not vibes

A vague pitch produces five polished versions of "needs more detail." That burns tokens and teaches nothing. Gavel evaluates readiness before it spends a model call.

The gate checks structure first: audience, problem, solution, pricing hypothesis, risky assumption, existing evidence, and either a current workaround or named competitors. Problem and solution must differ. Human evidence for a ready label is narrow: interview quotes, LOIs, payments, usage, or experiment metrics. A founder note does not count. After a completed run, another run needs new human evidence or a changed worksheet. Override exists. Default path asks for better input.

I kept this in ordinary code so the policy can be wrong in a reviewable way. Too strict or too loose is a PR with tests. It is not another prompt that drifts when the provider updates weights.

Contracts for judgment quality

Each judge returns a structured verdict: score, label, critique, key concern, recommended fix, and evidence that would change the score. The panel schema requires exactly one verdict per role. Schema validation catches shape failures. It does not catch five roles saying the same thing in different fonts.

A second layer checks score-label alignment, missing action fields, duplicate concerns, generic evidence requests, and heavy text overlap. Weak panels retry up to twice with role-specific instructions. Uniform panels can complete with a low-confidence flag. Non-uniform panels whose lenses still collapse onto the same proof fail the run. Those are different failure modes. Bland consensus is usable with a warning. Fake specialization is not.

Re-voting is capped at three points, and a score change must update its reason from the debate. Without that, the transcript is theater. Structural evals for all five roles and a monthly model-graded audit sit on top, because valid JSON is not useful judgment.

Streaming is not a run engine

Long model calls turn ordinary browser behavior into correctness bugs. People refresh. They close a laptop. They open the same run in a second tab. If the pipeline dies with the HTTP connection, you do not have an application. You have a fragile live demo.

Each run is one background task. Events append to SQLite with a monotonic sequence. SSE clients subscribe, replay, and reconnect with Last-Event-ID. The pipeline continues with zero listeners. The frontend reducer ignores stale or duplicate sequences, merges debate token deltas, applies re-votes, and converges to the same view whether events arrived live or after a reconnect. Cancellation, a wall-clock budget, and stale-run recovery after process restart live in the same manager.

One honest footnote: this is perceived streaming. The panel finishes all five calls, including retries, before emitting verdict events in display order. Debate is collected before custom token chunks reach SSE. Progress and replay are real. Provider-to-browser token forwarding is not what I built.

Pin every score to its input

Comparing two scores without knowing which worksheet produced them is cargo cult analytics. Gavel stores worksheet versions and links every judge run to the exact version it saw.

Core-field edits create a new version. Typos and non-core changes can stay put until that version is referenced by a run. After that, saves copy-on-write so history cannot be rewritten under an old score. Every save carries a base version id so the API can reject stale concurrent edits. Field diffs are string equality. A local workbench does not need embeddings to notice that pricing changed.

Critique has to become work

The post-run handoff is the feature that made the rest worthwhile. Synthesis problems become assumption drafts. Evidence requests become evidence targets. Recommended fixes become experiment drafts. The server stores at most twelve deduplicated suggestions. The UI asks the founder which ones belong in the ledger.

Calling them drafts is load-bearing language. Judge output is not evidence. An AI experiment is not automatically a good experiment. The system shortens the distance from critique to action without treating the model as the source of truth. Appeals stay separate: a founder can challenge selected concerns with new evidence, and the appeal re-evaluates the panel instead of writing validation records.

The validation ledger is the real surface

Once handoffs exist, the worksheet stops being a form. Assumptions have status. Evidence attaches to claims. Experiments carry hypothesis, method, and outcome. Interview notes sit next to the quotes that actually move readiness.

That feedback changed the judge schema. Every verdict asks for evidence that would change the score because that field has somewhere to land. Every synthesis problem is a candidate assumption because the workspace already knows how to track one. Primary-action routing after a run points at the next useful move: collect missing evidence, run a drafted experiment, revise the worksheet, or challenge a concern. Some of that routing is still mirrored in the frontend. The intent is not: leave people with a PDF of opinions.

Context without transcript landfill

A run can start with optional Tavily research, emitted before panel events so the UI can show what influenced the result. Past ideas can contribute memory by recency, or by semantic retrieval when that mode is on. Prompts never receive full old transcripts. They get compact records: scores, concerns, synthesis, appeal outcome. Local models drift when the window fills with prior arguments, and full transcripts are expensive even when the provider can absorb them.

Idea text, research, memory, worksheet fields, debate, and appeals are wrapped as tagged untrusted data. Interior tags are escaped. That does not erase prompt injection. It gives every model call the same explicit trust boundary, which is the best I can do without pretending delimiters are magic.

AI where ambiguity helps

Outside the judge run, Gavel can draft a worksheet from rough notes, clarify a field, suggest interview questions, summarize a week, propose assumptions, and scan competitors. Those are assists with a human in the loop. Model calls draft and judge. Regular code owns readiness, speaker order, event sequence, lifecycle transitions, and completed status.

I care more about that split than about whether the panel looks impressive in a GIF. Ambiguity belongs in judgment and drafting. Determinism belongs in whether the system is allowed to spend tokens and how the result is stored.

Local-first constraints

Gavel targets local or self-hosted use. No accounts. A stable local user id is enough for one machine. Judge runs can use Ollama. Some assist endpoints still assume DeepSeek, which is a hole I still need to close for a genuinely local-only path.

There are three SQLite databases: runs, workspaces, and idea memory. Different lifecycles made that convenient while the product was still moving. Cross-store consistency is the tax. The single-worker assumption is similar. In-memory task handles and subscriber wake-ups are simple when one process owns the world. They become wrong behind two API workers. The event log is already durable; multi-worker needs a shared queue and pub/sub.

Tradeoffs I would revisit

What stuck

The panel is still the part people want to see. Most of my debugging time went elsewhere: forcing a repeated run to require a real delta, keeping five roles distinct, capping re-vote movement, replaying ordered events after a reconnect, and stopping a typo fix from rewriting the worksheet under an old score.

That is the distance between the first prototype and Gavel. The model response used to be the last screen. Now it produces a short list of things a founder can test, ignore, or challenge before revising the worksheet and running the next version. The interesting engineering was making that loop honest.