KarstenSamaschke
Zurück zu allen Artikeln

Notizen aus der Praxis

A Silent Supervisor Is Not a Yes

When the supervisor model that checks an agent's "completed", "blocked" or "waiting" claim gave no usable answer, the Hayate Agent kept that claim in several cases. PR #232 keeps the task open in those cases instead, and the fix's twelve new test cases fail on the code before the merge and pass after it.

Karsten Samaschke · · 8 Min. Lesezeit

A Silent Supervisor Is Not a Yes

冬近し
答えなき夜
灯は消さず

Winter draws near.
A night without an answer;
the light stays on.

In my Hayate Agent (formerly the VanillaCore Agent), the working model is not supposed to get the last word on its own work. When it says a task is finished, blocked or waiting, a second model, the supervisor, reads the result and decides whether that claim holds. Picture the agent being asked to run an isolated test, with deployment explicitly not a dependency; this is the case the fix's own test builds. It plans the test, takes one turn, then reports "Deployment pending" and says it is waiting. The supervisor is supposed to catch exactly that. The awkward question is what happens when the supervisor does not answer at all.

Until pull request #232 merged this morning at 07:47, the answer in several cases was that the agent's own claim stood. I think that is the wrong default for anything that runs unattended. A missing check is missing evidence, and missing evidence must not turn a model's "I'm done" into the recorded outcome of the task.

Hayate Agent cover: small slate 疾風 logo and "Hayate Agent" at the top, the red Japanese headline 監督者の沈黙は 承認ではない (a supervisor's silence is not approval), the white English headline "A Silent Supervisor Is Not a Yes" and github.com/hayate-agent at the bottom, on an indigo-black field with a faint wave pattern along the lower edge

Where the old code let the claim through

The supervisor check runs after an eligible turn and either confirms the agent's proposed state or replaces it. If that check failed, the old code had one safety line, and it only ran when a supervisor was configured at all: it threw away an unassessed "completed" or "blocked" claim, but only on the first step of a task (continuation depth 0). That guard left gaps. A "waiting" claim was never covered by that line, at any depth. Once the supervisor had admitted a follow-up step, a "completed" or "blocked" claim from that later step was outside the guard as well. Without a configured supervisor, the line did not run at all.

The scenario above lands exactly in both holes: the first turn earns an admitted follow-up, the second turn claims to wait for a deployment that was never a dependency. If the supervisor then fails, by being unconfigured, returning nothing, throwing an error or answering with an invalid plan, the old code recorded the task as waiting. The task now said it was waiting on a deployment because the model said so, and nobody independent had looked.

PR #232 moves the guard. For an eligible autonomous turn, at any depth and with or without a configured supervisor, a completed, blocked or waiting claim that the supervisor did not independently confirm no longer becomes the task's state. The eligibility gates are the usual ones: the task is under its repetition limit and inside its deadline, and nothing is paused, stopping or queued. In the tested cases the task stays active, without a next step and without new work being queued, and it gets a small decision receipt saved with the task: what the model proposed, the verdict "unresolved", and, where the check reported one, why it gave no verdict (for example unavailable, technical failure or invalid response).

What a scripted run shows

The AI agent that drafts these changelog posts ran the PR's tests against the code on both sides of the merge, with Node 22 on a local machine. PR #232 adds one test file and changes thirteen other test files. The new file drives the real harness class with a fake model runtime and a scripted supervisor, and it covers the three claims (waiting, blocked, completed) against the four failure kinds (supervisor missing, empty answer, error, invalid answer).

Code New test file (12 cases) All 14 test files touched by #232 (81 tests)
Before #232 (9fa53c7, tag v0.1.156) 0 pass, 12 fail 68 pass, 13 fail
After #232 (25fbda7, tag v0.1.157) 12 pass 81 pass

All twelve failures before the merge stop at the same assertion: the test expects the task to remain active, and the old code had already written the model's claim into the task state, so it reads "waiting", "blocked" or "completed" respectively. The thirteenth failure is an older budget test that the PR rewrote: once an unassessed blocker no longer counts as proof, the host itself has to seal an exhausted repetition budget, and the test now expects that. These are controlled local tests with a fake runtime, not a live system. The PR author reports a full suite of 1287 of 1287 passing tests and an independent review with zero findings; that is the author's report, and the run described here covered only the 14 test files.

The same test also restarts the harness from the saved state with a working supervisor. The receipt survives the restart unchanged, and exactly one recovery step gets queued. In the code, every step admitted this way starts with the instruction to first reconcile the actual task state and completed side effects instead of blindly replaying them. From code reading, not from the run: if no supervisor is configured at all, the retained task is parked as blocked with the note that independent supervisor admission is unavailable, which is a host statement about the missing check, not the model's claim.

Why an open task beats a false done

The cost of this change is visible. A task whose check failed now sits there active and idle, waiting for a later independent assessment instead of looking finished. Some people will call that a stall; I call it honest. A task that says "completed" while the actual work never ran is worse than a task that says "not decided", because the false one gets closed, forgotten and trusted.

The night's other big supervisor change points in the same direction. PR #227, merged at 22:32, made the model's "continue" a proposal as well: only an affirmative supervisor verdict admits productive follow-up work, and absence, failure or an invalid answer hold the work instead of queuing the model's plan. #227 closed the door for unassessed work; #232 closes it for unassessed endings. The PR text itself draws the boundary: it tests the mechanics, not whether the supervisor model judges well, and it says nothing about any historical incident.

The rest of the window

The window ran from 20:00 on November 2 to 08:00 on November 3, Europe/Berlin time, and all times in this post use that zone. Core merged eight pull requests in it, the plugins repository three.

  • #225 (20:25) corrected the public installation and publication docs.
  • #226 (21:58) added Main-managed durable worker profiles and tasks.
  • #228 (22:41) bounds the supervisor's final prompt and marks whether the transcript evidence it saw was complete.
  • #229 (06:24) displays the model and reasoning level a worker actually ran with, next to the requested one.
  • #230 (06:32) persists opt-in model roles.
  • #231 (06:53) does the same observed-versus-requested split for the dispatcher's complexity classifier.
  • Plugins #41 (22:33), #42 (06:15) and #43 (06:45) are the companion changes: deadline and host evidence for the completion check, and the separate display of observed execution and classifier selection.

The test counts in these PRs are the authors' own reports. Nobody reviewed them independently for this post, and none of it proves live activation.

What merged and what shipped

Release tags follow the merges one by one: v0.1.150 is the #225 merge, v0.1.151 #226, v0.1.152 #227, v0.1.153 #228, v0.1.154 #229, v0.1.155 #230 and v0.1.156 #231. A git ancestry check confirms that v0.1.156 does not contain #232. The fix is first in v0.1.157, whose tag commit is the #232 merge; that release was published at 08:00:13, thirteen seconds after the window closed. A tag is a build, not a rollout: whether any running Hayate Agent uses #232 is not verified here. When the window closed, core main was at 25fbda7 and plugins main at 866480c; the next core merge, #233, landed at 08:06. The public GitHub snapshots under hayate-agent lag the upstream repository and were not checked for this change.

Verdict

A supervisor exists because the model's own report is not evidence. When the supervisor goes quiet, the only consistent answer is to keep the question open, and since #232 the Hayate Agent does that for endings and waits too, at any depth of an eligible task.

Sources used