Back to all writing

Field notes

The Pause Nobody Was Watching

A delegated job in my Hayate Agent ran out of checking rounds, was marked as paused, and the task waiting for it kept waiting with nobody in charge. Two fixes on Thursday give that kind of pause an owner without restarting the job, and keep a busy chat from starving that owner.

The Pause Nobody Was Watching

三たび「済み」
答えは「まだ」と
冬隣

Three times: "done."
The answer: "not yet."
Winter next door.

On Thursday, a delegated job in my Hayate Agent (formerly the VanillaCore Agent) stopped itself and then sat there. The job ran in steps. For its first step, the worker reported that the step was complete. An independent checker, a second model that has to confirm each step before the next one may start, answered that the step was not finished and asked for more. That happened three times: the first answer and two extra rounds. After the second extra round the step logic gave up, as it is designed to, and the job was marked as paused. The task that had handed out the job was waiting for its result, and it kept waiting. Nobody had paused the job on purpose, and no part of the host took responsibility for it: no assessment, no next step, no notice.

I think this is one of the nastiest failures an agent can have, because it looks like a decision. "Paused" normally means somebody chose to stop: an operator, a question waiting for an answer, an approval. Here it meant that a limit had run out and the result landed in a state the supervisor never looked at. Resuming such a job automatically would be the wrong fix. The worker's effects from that step were already settled, the checker had disagreed with it three times, and a rerun could repeat effects that had already happened. The right fix gives the pause an owner, not a restart. Pull request #276 does that, and #278, merged 82 minutes later, makes sure the owner still gets its turn when the same chat keeps the agent busy with other messages.

Hayate Agent cover: small slate 疾風 logo and "Hayate Agent" at the top, the red two-line Japanese headline 誰も見ていなかった / 一時停止 (the pause nobody was watching), the white English headline "The Pause Nobody Was Watching" and github.com/hayate-agent at the bottom, on an indigo-black field with a faint wave pattern along the lower edge

All times are Europe/Berlin. The window is 2026-11-05 08:00 to 20:00.

Why the pause had no owner

The limit the job ran into is only a few hours old. Until #271, merged at 12:55, any checker verdict other than "completed" turned into a generic block. #271 lets a checker that answers "continue" send the worker back into the same step, in the same session, for at most two extra rounds, with the checker's hint passed along as untrusted data. That is sensible. What nobody had wired up was the exit when both extra rounds are used.

The pull request description traces the live execution, d2494ffb, through the state files, the error log and the session transcript. When the strategy ran out of rounds, it failed the job. The record of an unfinished effect had already been cleared, so the job was saved with the generic diagnostic runtime-failure and the status paused, not blocked.

That one word mattered. The host already has a supervision path for a parent whose delegated children are stuck. A tool-free assessor, again a separate model, gets the original mandate and the children's status, and may propose one different, independent next step for the parent. It is told not to retry, resume or cancel the children. Otherwise the parent is marked blocked. The claim on that assessment is saved before the model is asked, and an assessment that fails technically is retried, with at most three attempts in total. That path only accepted children with the status blocked. The parent's own wait recheck was skipped as well, because the child still counted as pending. So the parent waited on a child that no code path would ever look at again.

What #276 changed

The fix is a 12-line change in src/harness.ts. A new check, supervisable(), accepts blocked children as before, and additionally a paused child only when all of these hold: the diagnostic is runtime-failure, no effect is still marked as in progress, no question is open and the parent was not stopped. The assessment stays read-only. It neither resumes nor replays the child. A child whose effect is still open never reaches this new branch anyway: a failure in that state already ends as blocked, which the existing path handles, and the child stays fenced there as well.

Why the child is paused What the new branch in #276 does
An operator paused it nothing, the pause stays binding
It asked a question that is still open nothing, the question stays binding
The parent was stopped nothing
Internal failure with settled effects, for example the exhausted checker rounds one independent assessment of the parent, retried only after technical failures, at most three attempts

The result of that assessment is either one new, independent step for the parent within its original authority, or a blocked parent. Here it matters what a person actually sees. In the code at this commit, a parent blocked this way, with a child that belongs to the task like d2494ffb, produces no chat message by default. The notice goes out only when the gateway has recovery diagnostics switched on, which is off by default, or when all three assessment attempts are used up. Older jobs without a task assignment take a separate path, which does send a notice. Either way the decision is stored with the task, and the child itself stays paused until someone acts on it.

The pull request also adds a regression test that replays d2494ffb: a real dispatcher and the step strategy drive a job into the same paused state with the same diagnostic shape, then a restarted host has to pick it up. A second test checks that a manual pause and an open question still get no assessment.

The second gap: a busy agent never got around to it

#276 had a limit of its own. The supervision path ran only when the conversation's session was completely idle: no turn in progress, nothing queued, nothing running. In a chat that keeps receiving unrelated messages, that moment may never come.

#278 adds a second place where the check can happen. After a foreground turn, meaning a chat or API message, has completed and before the next queued turn starts, the host assesses at most one waiting parent. It never runs that assessment at the same time as another model turn in the session. It skips a parent whose own input is already queued, and it skips a parent whose three assessment attempts are used up.

The tests show both sides of that change. In one, twenty consecutive foreground turns arrive, each held until the next one is queued. The parent gets exactly one assessment, at the first completed boundary, never more than one model turn runs at a time, and the child's record does not change. In another, the same boundary is checked against five holds: a manual pause, an operator stop, an open approval question, an exhausted assessment budget and a queued input for the same parent. None of them gets an assessment. A third test documents what is still true: a restored inbox that has not been released yet keeps the check waiting until that queue is released and drained.

The pull request's commit list shows how this came together. Commit 68827f4 at 16:46, five minutes after #276 merged, adds a multi-step test: step one completes, step two pauses after an invalid checker answer, and after a restart the parent gets exactly one assessment while step one is not repeated. The fix for the busy-chat case, fb595ae, follows at 18:00, and the merge at 18:03.

One message, one progress bubble

The evening's other change is about noise. Matrix can show the agent's live progress as native messages, an opt-in feature switched on with MATRIX_NATIVE_ACTIVITY=true. Before this change, one message to the agent could leave several persistent progress messages in the room, because the main turn, the dispatcher's classifier and the worker each posted their own. In the author's test, one input produced four progress messages.

Plugins #49, merged at 19:34, keys the progress message by the originating input instead of by each internal turn. Each internal turn becomes one line in that single message, with at most eight lines per message. Tool names and states stay visible, their arguments and results stay out, as before. Without the origin information, it falls back to one message per delegated job. After a restart, every tool line that was still running in a shared message is marked as interrupted. Core #280, merged two minutes later, passes the origin's ID through the dispatcher to the channel, without the message text or tags, and pins the plugins to that merge. The #49 description states plainly that it goes live only after that pin, a release and a restart of the agent. The pin and release v0.1.190 now exist. Whether any running agent was restarted on it is not verified here.

What a scripted run shows

The AI agent that drafts these changelog posts ran the test files added or changed by each pull request against the code directly before and after its merge, with Node 22 on a local machine. These are controlled local tests with fake model runtimes and temporary state, not a live system.

Change and tests Before the merge After the merge
#276 paused-child supervision, 2 tests 1 pass, 1 fails (699967e) 2 pass (979b625)
#278 boundary tests and the extended step test file, 37 tests 36 pass, 1 fails (979b625) 37 pass (09e450b)
#280 activity origin ID, 1 test 1 fails (v0.1.189) 1 pass (v0.1.190)

The failing test before #276 is the d2494ffb replay. It fails with the message that the paused child was orphaned: no assessment, no continuation, no notice. Before #278, the twenty-turn test gets zero assessments instead of one. At the last merge of the window, 065683c, the four files together pass 40 of 40. The plugins test for #49 was not run here. The authors report full-suite runs of 1,509 passing tests with 3 skipped for #276 and 1,517 passing with 3 skipped for #278.

The rest of the day

Several earlier merges deal with the same machinery. #268 at 11:19 saves a status update first and delivers it in the background, so a failed chat notification, including a Matrix rate limit, no longer stops work that was already authorized. #269 at 11:40 keeps unfinished older jobs that carry no task assignment from blocking tasks created after them. #273 at 14:44 gives the step checker 120 seconds instead of 10, and keeps the checker's diagnostic with the step. #275 at 16:34 removes an unconditional review obligation from the core defaults and states that such behaviour belongs in plugins. #263 at 08:01 lets an operator's natural-language budget reset accept Matrix task IDs that start with $.

#274 at 16:02 is described as a fix for test timing and teardown, but its commit also changes the step logic: a worker's own "blocked" report now goes to the checker as well, a "continue" verdict sends the worker into another round of the same step, and a "completed" verdict never overrides the worker's blocked report. #270, #272 and #279 change only tests, and #279 unblocks the Linux image build.

What merged and what was tagged

When the window closed, core main was at 065683c and plugins main at 7b41019. Each release tag sits on a merge commit: v0.1.185 on #263, v0.1.186 on #268, v0.1.187 on #272 and also contains #269, #270 and #271, v0.1.188 on #273, v0.1.189 on #279 and also contains #274, #275, #276 and #278, and v0.1.190 on #280 with the plugins pin. A tag marks the source of a published image, and a published image is not a deployment. The description of #278 calls #276 an already deployed fix, which is the author's statement and not verified here. The deployment repository had no merges in the window.

Sources