KarstenSamaschke
Zurück zu allen Artikeln

Notizen aus der Praxis

The Night Shift Opened a Pull Request

Until this morning, my agent harness added a run's token usage and cost without validating them, so a single NaN field could corrupt the totals. The fix came out of one night of its improvement loop, and the 37 tests that come with it all fail on the code before the merge and all pass at the merge.

Karsten Samaschke · · 7 Min. Lesezeit

The Night Shift Opened a Pull Request

On the night of 28 October, the nightly improvement loop in my agent harness logged 52 experiments, and the agent marked 45 of them as improved. To me, a self-graded "improved" is a note, not a change. It becomes a change when it arrives as a pull request with tests that someone else can run, and at 05:19 Berlin time this morning, two of those notes did exactly that.

What they fix is small and nasty. Whenever a run in the Harness finishes an assistant message, the runtime reports its usage: four token counts, a cost and the model name, and the Harness adds them to the run's totals. Until this morning it added them without looking. An update with a single broken number, for example input tokens set to NaN, went straight through: the run's input count became NaN, the cost from that same broken update was booked, the run took over the model name the update carried, and a fresh usage event went out to everyone listening. Nothing was rejected, and nothing pointed at the bad field.

I like this fix more than its size suggests, and the bug is not the reason. Core pull request #153 is a handover from the night shift that I can check instead of taking it on faith. Its description calls it the first explicitly labeled nightly proposal, traces it back to records 8 and 9 of that night, and says outright that the original "improved" receipts are provenance, not an independent pass. The agent opened it on 29 October, and it was merged about 46 hours later.

A closed dark laptop behind a tall stack of blank white cards on a dark navy desk, with one single card lying apart in front, edged in pale light blue. Text: An AI agent's night shift claimed 45 improvements. Two became one merged, tested fix.

What changed

#153 checks the whole update before it touches anything. All four token counts must be non-negative whole numbers within JavaScript's safe integer range, and the totals they would produce must stay within that range. The cost must be a finite, non-negative number, and the cost total must stay finite. If any of that fails, the Harness drops the entire update: no token counts, no cost, no model name and no event. Valid updates after it go through as before, and a zero-token update or a fractional cost is still accepted. The change to the source is seven added lines in src/api/runs.ts; the other 81 added lines are tests.

The pull request lists its own limits, and they matter. It repairs no record that is already damaged, it adds no general event decoder, and a dropped update leaves no diagnostic behind. A runtime that keeps sending broken numbers now produces a run whose totals silently miss those messages, and nobody is told. The cost field is also mandatory now: an update with a missing, null or text cost counts as invalid and is dropped whole. I would rather have a quiet gap than a NaN in a cost total, but I want those drops counted somewhere, and that part is still open.

What a scripted run shows

For this post, I had the three test files from #153 run on the commit right before the merge, 1982550b, and on the merge itself, 170b06b8. Before the fix, all 37 tests failed. At the merge, all 37 passed.

The failure output shows the problem better than any description. In one test, the run has already counted a valid update with 1 input token, 2 output tokens and a cost of 0.1. Then it receives an update with input set to NaN, a cost of 3 and a different model name. Before the fix, the run ended up with input tokens NaN, a cost of 3.1, the model changed/model and its event sequence advanced by one. After the fix, the run looks exactly as it did before the bad update. Another test sends a cost that pushes the total past the largest number JavaScript can represent, and before the fix the stored cost became Infinity. These are controlled local tests against the real run registry class with a temporary state file, not a running agent.

The rest comes from the agent's own report in the pull request. Its focused set of 46 tests passed, and typecheck and build succeeded. The full suite did not pass: under a 300-second timeout it stopped with 683 passed, 3 failed and 49 cancelled, and the description traces the three failures to problems it says also show up on the base. An independent review of the exact commit found no open P0 or P1 findings. The Harness's review plugin was still tied to an unrelated change, and the pull request says plainly that it claims no pass from it.

The rest of the night

The same agent account, severus-snape, merged seven more pull requests into core between 04:11 and 06:06. #168 adds a targeted test run that follows relative imports to pick the tests a change probably touches, can pick too many and falls back to the full suite for unknown or uncovered files and a list of shared core files. #169 deals with 15 test failures that blocked the image build, partly by updating outdated test expectations. #170 keeps an operator message that arrives during an active turn as a task of its own, so an answer to something else cannot complete it. #171 skips an oversized skill with a size notice instead of blocking the turn. #172 adds a default instruction for the agents, one purpose per pull request and small, independently deployable increments, and the code calls it an instruction, not enforcement. #173 and #174 only add diagnostics around a child-process failure that kept blocking image delivery, and neither claims to fix it.

All seven were merged within eight minutes of being opened, five of them within 15 seconds, and each description cites a passed review of its exact tree. The night shift's proposal took the slow road: it was opened as "only a proposal, no merge, no deployment" and waited almost two days. Its merge commit carries the same agent account, and the pull request shows no formal review, so the public record does not say who decided to merge it.

What merged and what shipped

This post covers 30 October 20:00 to 31 October 08:00, Berlin time (CET), which is 19:00 to 07:00 UTC. Core merged eight pull requests in that window, #168 to #174 and #153, all opened by the agent account severus-snape. The plugins and deployment-base repositories merged nothing, and no release was published in the window.

All eight merges first appear in v0.1.114, published at 11:39 on 31 October, after the window. That release also contains #176, merged at 11:35, which limits how many processes and threads the targeted test runner runs at the same time, and v0.1.115 at 12:09 adds #175. A release tag is not a deployment, and nothing here says which version any installation runs.

Where I land

Forty-five self-reported improvements from one night told me very little. One pull request with seven lines of logic, 81 lines of tests and an honest list of what it does not do told me more, mainly that the night shift can produce something a person can check, with a diff, tests that fail before and pass after, and its limits written down. I would keep that bar exactly where it is and give the dropped updates a counter before the next night proposes something bigger. The earlier piece on the loop itself is The Night Shift That Grades Itself.

Sources used