KarstenSamaschke
Zurück zu allen Artikeln

Notizen aus der Praxis

Hayate Agent Counted Bytes and Called Them Tokens

A guard added on Saturday compared JSON bytes with a 131,072-token limit and turned away legitimate requests before any model saw them. Why #467 removes the gate instead of raising the number, how far bytes and tokens really are apart, and the cost question it leaves open.

Karsten Samaschke · · 10 Min. Lesezeit

Hayate Agent Counted Bytes and Called Them Tokens

冬の朝
バイトで量り
門の外

Winter morning.
Weighed in bytes,
outside the gate.

On Saturday, according to core pull request #467, Hayate Agent v0.1.325 in production was turning away requests from agents called Peter and Odo before any model saw them. Each refusal said the model input budget was exceeded, that the history was kept and that no request had gone to the provider. The description calls those requests legitimate, and it names the cause in one line: the size of each request was measured in bytes of JSON, and that byte count was compared with a limit of 131,072 that the code treated as tokens.

Hayate Agent cover on a dark indigo background with a faint wave pattern across the lower part: small slate 疾風 logo with the label Hayate Agent, then a two-line red Japanese headline, Hayate Agentは / バイトを数えてトークンと呼んだ, which means Hayate Agent counted bytes and called them tokens, the white English headline Hayate Agent Counted Bytes and Called Them Tokens, and github.com/hayate-agent at the bottom.

A token is the unit a model provider counts, bills and limits, and in ordinary text one token covers several bytes. For this post, four sample inputs were measured with that byte estimate and with OpenAI's o200k tokenizer, and at the 131,072 threshold they came to roughly 25,000 to 44,000 tokens, depending on the content, so the guard stopped work at a fraction of the size it claimed to enforce. The refusal also told the agent to compact its context, and the summary request that compaction sends was measured with the same ruler, so a summary request that was itself too large by that count was refused as well. I like the way #467 fixes this: it does not raise the number. It removes the hard gate, keeps the byte estimate as an early signal for compaction, and leaves the real limit to the provider, which knows its own tokenizer.

All times are Europe/Berlin. The window is 14 November, 20:00, to 15 November, 08:00.

A guard from the same afternoon

The gate was not old. Core #460 merged at 13:06 on Saturday and was first contained in release v0.1.321, published at 13:57. Among other storage work, it promised to "gate the final assembled PI model input against the byte-based admission budget, including tool responses". The reasoning behind it is sound and is spelled out in the code comment: no message is removed or shortened, so an oversized request fails closed instead of silently dropping one half of a tool call and its result, or the authority a task carries. The docs at the time also said plainly that this was "a message-size admission guard, not an exact provider tokenizer".

The trouble was the comparison. The estimate started at 256 units for framing and added the UTF-8 length of every message as JSON plus 32 units per message. It compared the total with a constant named MAX_MODEL_INPUT_TOKENS, set to 131,072, which was the ceiling; for a model with a small context window, the room reserved for output could lower it further. When the estimate was larger, the adapter returned the error without opening a provider stream. #465, merged at 16:21, extended the context-maintenance budget report to every model, counted its message history with the same estimate function and dropped a division by three that the status report had used to turn bytes into rough tokens. From v0.1.323 on, compaction pressure, maintenance and the final gate all counted bytes. They did not always measure the same messages or use the same threshold, but none of them counted tokens, and the final gate was the one that refused work.

What the 131,072 threshold holds in tokens

The AI agent that drafts these changelog posts serialized four kinds of input the way the gate did, one message each, and counted the same JSON with OpenAI's o200k_base tokenizer. The chart scales each measured ratio to the 131,072 units above which the gate refused a request.

Bar chart titled What the 131,072 threshold holds in tokens, subtitle the gate refused above 131,072 units of its byte estimate. Four white bars on dark tracks, each track is 131,072 tokens: English docs page about 25,531 tokens, TypeScript source file about 31,223, PR list as JSON about 37,358, #467 test fixture about 43,680. Footer: counted with OpenAI o200k_base, one message each; other tokenizers differ; source core PR 460 and 467.

An English page of the core documentation came out at 5.1 estimated bytes per token, a TypeScript source file at 4.2, a page of pull-request JSON from the Forgejo API at 3.5, and the test fixture from #467, 46,000 copies of the character 界, at 3.0. At the gate's limit, that is between about 25,500 tokens for the English page and about 43,700 for the fixture. The fixture is also the case where bytes come closest to tokens, because each of those characters takes three bytes in UTF-8 and about one token. Other providers use other tokenizers, so the exact ratios differ; the order of magnitude is the point, since none of the four came anywhere near 131,072 tokens.

What the fix keeps and what it gives up

#467 deletes the admission check from the function that hands every working and summary request to the provider. The byte estimate stays where it is useful: when the projected history passes 75 percent of the budget, Hayate Agent still triggers native compaction at a settled tool boundary, and nothing in the history is shortened or dropped. If a request really is too large, enforcement is now left to the provider or its SDK. A new test in #467 covers that path: a tool runs once, the provider refuses the oversized follow-up, and after the session is reopened the tool call and its receipt are still on disk and the tool has not run a second time.

The two test files from #467 were run against v0.1.325, the release before the fix, and against the merge. On v0.1.325 both tests fail with the old refusal, "Model input budget exceeded (142823 conservative units > 131072)" in the first and 141,464 units in the second, while the fake provider in the test declares a context window of 1,000,000 tokens. At the #467 merge both pass, and they still pass at the last merge of the window, #487. These are controlled local tests with a fake provider. The author reports a full local run of 2,587 passing tests with 1 skipped, and an independent review that passed with one note about cost; neither was rerun for this post.

That note is the price of the fix, and the description states it openly: Hayate Agent no longer has a host-side hard ceiling on input size, and a real resource or cost boundary "requires a future provider-aware token mechanism, not a byte=token comparison". I would rather run with a named, open cost question than with a gate that refuses legitimate work because it measures in the wrong unit. A byte ceiling does bound the serialized size of a request, but it does not enforce a 131,072-token budget, and in these samples it refused work at token counts that depended on the content.

The rest of the night

#468, merged at 22:00, continues Saturday's storage work on remounted volumes: after an fsGroup remount left an agent's task-book directory at the owner-private mode 2700, the startup check refused it, and it now accepts 2500 and 2700 while keeping the owner and symlink checks. #476 normalizes an archive search query once per read instead of once per entry; in the author's synthetic run with 500 entries, that is one normalization call instead of 500, with the same results.

The other eight merges came from the account peter-hayate-dev, which delivered fixes from its backlog of earlier night experiments, each with a new or changed test that fails before the fix and passes after it, as its descriptions report. #469 accepts leading JSON whitespace before double-encoded tool arguments, and #479 stops an apostrophe inside a word, as in "don't", from hiding later hashtags. #481 replaces bidirectional and line-separator characters in the path preview of an approval request with spaces, so a path cannot be displayed reversed or broken across lines. The configuration checks got stricter or more forgiving where the old behavior was wrong: #482 accepts SHA-256 pins in upper case and normalizes them, #483 rejects repeated or out-of-order interval units such as "5m5m" instead of adding them up, #484 rejects blank or space-padded cron job names, #485 rejects a review.scope that is not an object, and #487 rejects unknown keys under memory.embedding.

What merged and what shipped

In the window, 11 pull requests merged, all in core, and 9 core releases were published, v0.1.326 at 20:54 to v0.1.334 at 04:20. Plugins, webui and deployment-base merged nothing and published no release. Each tag resolves to the merge commit of one pull request: v0.1.326 is the #467 merge, published ten minutes after it, and v0.1.334 is the #487 merge. v0.1.332 is the #483 merge and is the first tag that contains #482, and v0.1.333 is the #485 merge and the first that contains #484. Fourteen core pull requests opened in the window were still open when checked, among them #491, opened at 04:22; none of them is covered here. A release tag identifies the tagged source commit; it says nothing about which installations run it. When core was fetched at 08:03 on Sunday, main was at the #487 merge, and a second fetch before handoff showed no change.

Limits

The production refusals of Peter's and Odo's requests and the review result are the author's reports; this post did not see either agent. The byte-to-token figures come from one tokenizer and four sample inputs, not from production traffic. Compaction pressure is still measured in bytes, so compaction can start earlier than the token count would require, and the status report still shows the byte estimate in a field named contextEstimateTokens. With the gate gone, Hayate Agent hands every request to the provider or SDK stream and leaves the input limit to that layer, so an oversized request is no longer stopped on the host before that call.

Verdict

A fail-closed guard is a good idea for a request that carries authority and unfinished work, and #460 had the right instinct about what must never be cut. A guard that stops normal work is only as good as its unit, and a byte count compared with a token number stopped requests at a fifth to a third of the size it was meant to allow, in the samples above. #467 takes the guard out instead of tuning it, keeps the estimate for the job it can do, and writes the missing cost boundary down as open work. That is the honest order: first stop refusing legitimate work, then build a limit that counts what the provider counts.

Sources used