KarstenSamaschke
Zurück zu allen Artikeln

Notizen aus der Praxis

Why Hayate Agent's Saved Tasks Did Not Resume

Hayate Agent keeps unfinished tasks on disk, yet real saved tasks did not resume after restarts. Each one has to pass a judge first, and six blockers around that judge made the tasks wait without anyone saying no. Here is what #433 and #435 changed, and what they still do not prove.

Karsten Samaschke · · 10 Min. Lesezeit

Why Hayate Agent's Saved Tasks Did Not Resume

冬の朝
判定を待つ
門の前

Winter morning.
Waiting for the verdict
before the gate.

Hayate Agent keeps unfinished tasks on disk, so a restart should not erase the work. A saved task that has no recorded permission to continue is first sent to a judge, the configured completion-check provider, together with its retained report and evidence. The judge decides whether it may go on, is blocked or is done. The default judge is a separate model review; other providers, such as the TypeSafe plugin, run without a host model. A task that already had its permission recorded is rechecked and can start without another judgment. Earlier restart proofs had passed with synthetic tasks. Core pull request #433 opens with the report that undid that comfort: "real persisted tasks did not resume despite prior synthetic restart proofs." The fix that followed is a useful lesson in how a careful safety check can stall saved work without ever saying no.

Hayate Agent cover on a dark indigo background with a faint wave pattern at the bottom: small slate 疾風 logo with the label Hayate Agent, then a two-line red Japanese headline, Hayate Agentで保存済みの / タスクが再開しなかった理由 (why Hayate Agent's saved tasks did not resume), above the white English headline Why Hayate Agent's Saved Tasks Did Not Resume, and github.com/hayate-agent at the bottom

All times are Europe/Berlin. The window is 13 November, 20:00, to 14 November, 08:00. #433 merged at 23:30, #435 at 00:13.

Six blockers around the judge

None of the problems in #433 was a judge saying "stop". The first three kept a usable verdict from coming back, or filed a valid uncertain answer as broken. The fourth let the judge time out before it had really started. The last two were scheduling: an inherited backoff delayed the next attempt, and the startup scan held back tasks that already had permission. In every case the task simply waited. The author's diagnosis, from installed agents, lists the reasons. I have not checked those installations, so the numbers below are the author's.

The first was size. The host builds the judge's input from the task's original mandate and its recent evidence, and on real tasks that input was 24 to 53 KiB. The TypeSafe judge plugin accepted at most 24,000 bytes and rejected anything larger without saying why. Plugins pull request #101 raises that limit to 56,000 bytes, the same envelope the host already used, and lets a judge declare its own limit as maxPromptBytes (1,024 to 56,000). The host then trims the observational evidence to fit both limits. The original mandate and its expiry are never cut to make room; if they do not fit, the judgment is unavailable instead of quietly based on half the task.

The second was a name. The configured model was the alias openjev-latest, and the gateway answered as openjev-0.1, the model the alias resolves to. The plugin required an exact match and threw the answer away. It now accepts a well-formed resolved name, while the endpoint, model and credential of the request stay fixed. The third was doubt: an answer with confidence below 0.9 counted as malformed. It now counts as "uncertain". That still does not let the task continue, but it is no longer filed as a broken response. The 0.9 floor itself stays.

The fourth was time. The author measured a real reviewer at 32.69 seconds, including retries after HTTP 503 errors, against a 30-second default. The default is now 120 seconds, the same allowance a Dispatcher step check already had, and an explicitly configured limit is used exactly as set. The less obvious part: the clock now starts when the provider starts working, not while the request waits for a free slot on the host's shared model capacity. Before, a judge could time out in the queue without ever being asked. Stop, expiry and cancellation still apply while it waits.

The fifth was memory of failure. Every unusable judgment pushes the next attempt further out, from one minute up to one hour. After a fix, a task that failed under the old limits would still sit out its inherited backoff. #433 records a fingerprint of the judge's policy with each attempt: a hash of the provider name and version, its input limit, the timeout and the validator role settings. When that fingerprint changes, the task gets one fresh, read-only judgment of its existing report. Nothing is replayed and no evidence or work budget is reset; the fresh attempt still counts as one more assessment attempt. A restart with the same fingerprint keeps the normal backoff, so restarting cannot be used to skip it.

The sixth was a queue at startup. The recovery scan walked through all saved tasks and released the ones that already had permission to run only after every judgment in the scan had finished. One slow judge for an unrelated task held back all the others. They are now released as soon as they are admitted.

The readiness signal that waited for everyone

#435 is the residual the author found while checking #433 on an installed agent. The author reports that a real Matrix task on an agent called Akira resumed with the same task id and moved from depth 40 to 48. But the process itself still waited for the entire semantic scan before it announced that it was ready. With judgments that take 30 to 120 seconds each, a backlog could keep an otherwise healthy agent "not ready" for minutes. A code comment in #435 names the consequence for Kubernetes: the agent can fail its startup probes.

The fix moves only the judgments behind the readiness signal. Restoring saved state, reconciling stops, cancellations and resets, recovering Dispatcher work, authenticating channels and publishing the durable inbox all still happen before "ready". The judgments run afterwards through the same path, and each task still checks its own admission before it does anything. If the scan itself fails unexpectedly, the agent shuts down gracefully instead of continuing without its checks.

Infographic on a dark indigo background with a faint wave pattern at the bottom. Red headline Six recovery blockers around the judge, subtitle why saved tasks waited after a restart, and what #433 and plugins #101 changed. Six cards, each with the old behavior and the change. Too big: real input 24 to 53 KiB, plugin limit 24,000 bytes; now 56,000 bytes, mandate never cut. Different name: asked openjev-latest, answered openjev-0.1; resolved name accepted, request unchanged. Unsure: confidence below 0.9 filed as malformed; now uncertain, the task still waits. Too slow: reviewer took 32.69 s, limit was 30 s; now 120 s, counted from provider start, not queue. Old backoff: failed tasks kept waiting up to one hour after a fix; one fresh read-only judgment per changed judge policy. One slow judge: held every admitted task back during startup; admitted tasks now start right away. An outlined box below reads #435: ready no longer waits for the judges; restore, stops, channels and inbox finish first, judgments run after; each task still checks its own admission. Footer: Source: core #433, #435, plugins #101. Numbers are the PR authors' reports.

What the tests show

#435 ships a process test I like, because it starts the real built entry points, dist/cli.js and dist/launcher.js, with one saved task and a fixture judge that never answers. It passes if the agent says it is ready and answers an authenticated API request while that judgment is still open, and if a normal shutdown then cancels the judge. The AI agent that drafts these changelog posts ran that test on a local build at two commits:

Commit dist/cli.js dist/launcher.js
Before #435 (v0.1.304, #434 merge) fails, no readiness after 16 s fails, no readiness after 16 s
#435 merge (v0.1.305) passes passes

The test files #433 added or changed tell a simpler story. At the commit before #433 (v0.1.302) both files fail on import, because the new assessmentPolicyStamp export does not exist yet, so none of their tests run. At the #433 merge (v0.1.303) all 112 tests in them pass. These are controlled local tests with fake runtimes and fixture judges. They show the mechanism, not a live system.

The author reports 2,402 passing tests with one skip for the full suite of #435 and writes that #433 was installed and checked on all 13 agents with eight behavioral checks each. #433 itself is careful about what it does not prove: of ten real task scopes in a shadow run, eight were accepted and two stay held for a privacy export. The TypeSafe service's own 5-second HTTP budget timed out, and a longer diagnostic request returned HTTP 504 after 60.5 seconds. The author calls that a separate external latency problem and says final acceptance requires actual task progress on installed source. None of this was rerun for this post.

The rest of the night

A smaller fix fits the same theme. #450 lets unauthenticated HEAD requests to /health and /v1/health return 200 like GET. Before, uptime monitors, load balancer probes and curl -I got a 401 from a health endpoint that was meant to be public. Other endpoints still require authentication for HEAD. Its author is peter-hayate-dev, and the description says it came from a nightly experiment.

#441 adds small local prompt budgets and scoped working memory per step. Its description says the tests support no production activation or real model speed claim. Plugins #103 adds durable semantic turn summaries for memory, and its description claims no runtime activation.

What merged and what shipped

v0.1.303 resolves to 025276a, the #433 merge, published at 23:45. v0.1.305 resolves to f24b7fc, the #435 merge, published at 00:30 on 14 November. v0.1.314 is the #450 merge (04:48) and v0.1.315 is the #452 merge (05:38). A release tag identifies the tagged source commit; it says nothing about which installations run it.

In the window, 23 pull requests merged, 20 in core and 3 in plugins, and 19 core releases were published, from v0.1.297 at 20:02 to v0.1.315. When core was fetched on 14 November after the window closed, main was still at the #452 merge.

Limits

A faster judgment is not permission. A task with an uncertain, missing or late judgment still waits, and the 0.9 confidence floor, stop, expiry and ownership checks are unchanged. The author reports that the two task scopes held for a privacy export stayed held; this post did not rerun that check. The external judge service's latency is not fixed by these changes. Readiness now means the structural restore is done, not that every saved task has been judged.

Verdict

A check that fails closed is only safe if someone notices when it fails. Here six blockers around one judge all looked like waiting rather than refusing. #433 removes the documented size, name, timing and scheduling blockers, and #435 stops the judgments from holding back the agent's readiness, without letting any task skip its judgment. Whether the external judge service now answers in time on real installations is still open.

Sources used