KarstenSamaschke
Back to all writing

Field notes

Hayate Agent Crashed on Files That Were Fine

An installed agent kept crashing at startup over files that were intact; a volume remount had only made them group-accessible. How the fix keeps the ledger check strict, and the worse bug found on the way: a missing page that could start the agent with an empty ledger.

Karsten Samaschke · · 10 min read

Hayate Agent Crashed on Files That Were Fine

冬の宵
帳簿は無傷
閉じた門

Winter evening.
The ledger is unharmed,
a closed gate.

An installed agent called Minerva stopped coming up. Kubernetes kept restarting it, and the error said that a part of its saved task ledger was missing or unreadable. Nothing was missing. A read-only audit found all 865 referenced files in place, every SHA-256 correct and every file owned by UID and GID 1000. What the audit did find was a permission: after the storage volume was mounted again, every file was readable and writable by the agent's group as well, mode 0660 instead of 0600.

Hayate Agent cover on a dark indigo background with a faint wave pattern across the lower part: small slate 疾風 logo with the label Hayate Agent, then a two-line red Japanese headline, 正常なファイルで / Hayate Agentが落ちた (Hayate Agent went down on files that were fine), above the white English headline Hayate Agent Crashed on Files That Were Fine, and github.com/hayate-agent at the bottom

That ledger is where Hayate Agent keeps the work it has accepted and the authority each task carries, so its reader is meant to be paranoid. The hard part of the fix is that both quick answers are wrong. Accepting any file the reader finds would turn a safety check into decoration, and editing permissions on a production volume by hand only lasts until the next mount. I like that core pull request #466 does neither, and I like even more what its author found on the way: in one of the stores, a page that really was missing could make the agent start with an empty ledger.

All times are Europe/Berlin. The window is 14 November, 08:00 to 20:00.

One bit the new reader did not expect

Kubernetes has a pod setting called fsGroup. For volume types that support it, Kubernetes by default changes the ownership and permissions of the volume's contents to match that group when the volume is mounted (some CSI drivers do this themselves), and the container's processes get the group as a supplementary group. An agent that seals its files to 0600 gets 0660 back after a remount. Hayate Agent already knew this. The execution-control directory has a code comment saying that fsGroup can re-add group permissions on every remount, and the dependency store that #457 added at 11:13 accepts group read and write as long as other users get nothing and nothing is executable.

The ledger partitions came later in the day. #460, merged at 13:06, split the execution ledger into immutable, hash-addressed files so a change writes only the parts that changed. Its reader demanded exactly 0600. The #466 description names that mismatch as the root cause: the new reader was stricter than the existing one next to it. It also refused with a misleading message. A scripted run at the #464 merge (v0.1.324, the commit before the fix) wrote one partition, set it to 0660 and loaded it again. The load failed with "Missing/unreadable partition", for a file that was present and readable.

The new rule is narrow. A partition file must be a regular file and not a symlink, owned by the agent's user, and either 0600 or 0640/0660 with a group the running process actually belongs to, either as its primary group or as a supplementary one. A group-readable file whose group the process is not in is still refused, and so are executable, world-readable and special modes. Loading stays read-only: after a load at the #466 merge, the scripted run's file was still 0660. When the normal durable writer re-verifies an existing file, it sets it back to 0600 through the descriptor it already holds and syncs it. In one place the rule is now stricter than before. The old check masked off the setuid, setgid and sticky bits, so files with modes 1600, 2600 and 4600 passed it; at the #466 merge they are refused.

The description is also clear about what the fix is not. It changes no authority, recovery, task, model or budget logic, it replays nothing, and it says the fix reaches agents through the normal image publication and GitOps path, never through a chmod in production or an edit to the ledger.

The empty ledger behind the first fix

The first commit of #466, at 17:43, fixed only the partitions. Eight minutes later the second commit extended the fix to the task book, the store #462 added at 13:41 for operator-owned tasks. The author's reason was concrete: the agent's current task book also sat in a directory and pages that the remount had made group-accessible, 2770 and 0660. Fixing the partitions alone would have moved the crash one store further.

The more serious finding came with it. When the ledger loader hit a "file not found" error, it treated the ledger as new and empty. That rule exists for the very first start of an agent, when no ledger file exists yet. With the task book, a committed page that was really gone raised the same error from deeper inside the load, so the agent would start as if it had never accepted any work. The author writes that an incomplete local test fixture without task-book pages exposed this, that it was fixed before anything was activated and that it is no acceptance receipt.

The AI agent that drafts these changelog posts reproduced both sides with a throwaway test against the real classes: save one execution and one enrolled task, delete the task-book pages, load the ledger cold, then write once more.

Step Before #466 At #466
Group-readable partition load refused loaded, mode untouched
Task-book pages deleted 0 executions load refused
One new write, reload old work gone not reached

Before the fix, the cold load reported no executions at all, and after one new submission the reloaded ledger held only that new request; the saved one was gone from it. At the #466 merge the cold load fails with "Cannot load retained task book pages" instead. This is a controlled local run with a fake request, not a production log.

What the tests show

#466 adds one test file with 12 tests. The same file was run against the commit before the fix and against the merge, each in its own installed tree:

Commit Passing Failing
#464 merge (v0.1.324) 4 8
#466 merge (v0.1.325) 12 0

The four that already passed before are rejections of modes 604, 664, 700 and 760, which the old reader refused as well. The eight that fail before cover the 0640 and 0660 cases, the supplementary group, the task-book remount and missing pages, the three special-bit modes the old check let through, and a test that checksum and symlink rejection still hold. The description reports 11 new regressions plus the existing partition and compatibility tests, 33 in total, passing, and an independent review that passed after the supplementary-group case was addressed. On a private copy of the real ledger, the author reports that a read-only load kept all 849 execution records and left the manifest hash unchanged. None of that was rerun for this post.

I trust this fix more because the incident report comes before the change. The description starts from the read-only audit (865 files present, hashes correct, owner correct), so the fix only had to explain one fact, the mode, and the tests pin exactly that fact.

The rest of the day

Five plugin merges and their core pins dealt with the TypeSafe judge. Plugins #104 and #105 stop TypeSafe timeouts and HTTP failures from being filed as malformed answers and set a 120-second default; core #455 pins that. Plugins #106 bounds the routine evidence sent to the judge without cutting the original task scope, pinned by core #461. Plugins #108 accepts the extra metadata OpenRouter returns with an answer, pinned by core #463.

On the storage side, #457 removed a failure where Minerva's ledger sat 18 bytes below the 32 MiB limit and writes and one container start failed with "Execution ledger size limit reached"; large dependency data now lives in immutable files outside the ledger. #460 partitioned the ledger and gates the final model input against the byte budget. #462 adds the operator-owned task ledger and outcome-bound plans, with plugins #107 as its validator counterpart. #465 starts context maintenance before the final input budget runs out. #464 binds local inference scheduling metadata at the HTTP transport and calls itself source support only, with no shared queue activated. #453 stops dispatcher loops on unchanged blockers, #454 pages skill metadata in worker worktrees, and #458 and #459 adjust how inconclusive and reopened judgments are handled.

What merged and what shipped

In the window, 18 pull requests merged, 13 in core and 5 in plugins, and 10 core releases were published, v0.1.316 at 10:39 to v0.1.325 at 18:12. deployment-base and webui merged nothing. Each release tag resolves to the merge commit of one pull request: v0.1.321 is the #462 merge and is the first tag that contains #460, and v0.1.325 is the #466 merge, published eleven minutes after it merged at 18:01. The task book arrived with #462, so the empty-ledger fallback described above concerns v0.1.321 to v0.1.324 and is closed in v0.1.325; the scripted run checked only v0.1.324. A release tag identifies the tagged source commit; it says nothing about which installations run it. When core was fetched at 20:03, main was still at the #466 merge.

Limits

The incident details, the 865-file audit, the 849 retained records and the review result are the author's reports; this post did not see the agent or its volume. The scripted runs use temporary directories on a Mac and the PR's own tests, not Kubernetes. Group access is accepted only for the agent's own files and only for groups the process belongs to, so a group-readable file whose group is not one of the process's groups still stops the agent. The missing-page fix makes the agent refuse to start; getting the pages back is a separate job, and #466 does not do it.

Verdict

A strict reader is right for a ledger that carries authority, but it has to be strict about the properties that matter: owner, group membership, file type and content hash. A single 0600 literal is the wrong anchor when the platform can rewrite it on a remount. #466 moves the check to those properties, and on the way it closes the worse hole, a missing page that read as an empty ledger. An agent that refuses to start can be fixed; an agent that starts having forgotten its work is much harder to notice.

Sources used