A developer asks a coding agent to fix a preview that sometimes appears in the wrong language. The preview cache keys entries by document ID, but rendering also depends on the locale. An English request can fill the cache entry that a later German request receives.

The agent finds the missing input, changes the key, and says the tests pass. Can you trust that last sentence? Did the edit run? Did the test use the changed file? Did another edit land after the test? Did the response end normally, or did the output-token limit cut off a command?

Answering those questions is the job of the agent harness, the code around the model. The harness runs the tool calls the model asks for, records what happened, and decides when the task is done. This post follows the harness's agent loop. The loop calls the model, runs the tool calls in its reply, and sends back the results until a stop rule fires. You will see which checks prevent a false “tests pass.”

The repository has a root test configuration, a preview package, and a nested locale package with its own instructions. Its working tree already holds an unrelated developer change. The agent must add the locale to the cache key, add a regression test, and keep that change.

The six-part Harness Engineering series uses the source-code study Harness Engineering as a map.

A tool call does nothing until the harness runs it

Start with a model response that contains two tool calls: read the preview code, and read the test configuration. A tool call is a structured request to run a named tool. No file has been opened yet. The model learns the contents only when the harness runs each call and returns a tool result.

Chat interfaces blur this difference. “I will inspect the cache” looks much like “I inspected the cache.”

To run one call, the harness must:

• resolve its paths against the real working directory;
• find the named tool;
• validate the arguments;
• check the permission rules;
• start the operation;
• attach the result to the call by its tool-call ID.

A failure at any step changes what the agent knows.

A missing file says something about a path. A denied read says something about permissions. Neither result shows that the code has no cache. If the harness turns both into an empty string, the next response starts from an invented fact.

So the agent loop carries more than conversation text. The loop records which calls each response asked for, which calls ran, and which result belongs to which call.

The agent loop has four kinds of done

The repair passes four points where something can be called “done.” Each answers a different question.

Suppose a test command exits with status zero. The process succeeded. It may still have selected no tests, run only a neighboring package, or checked a build artifact made before the edit. The harness should keep the command and its output so the agent can tell which.

Now suppose the model returns a text-only answer after changing the cache. The response ends with no new tool calls to run. Nothing yet shows that the regression test exists. The harness can treat the answer as the end of the run, or require another check before the agent reports completion. Either way, the harness needs an explicit rule.

The difference matters even for honest answers. “The implementation is updated; the regression test could not run because its dependency is missing” is a fair end to an attempt. “Fixed and verified” needs a test that passed on the final code. A single done bit cannot express both outcomes.

Tool calls from a cut-off response should fail

With streaming, the reply arrives piece by piece, and a tool call's arguments can be incomplete until the stream ends. The provider's stop reason (finish_reason or stop_reason, depending on the API) says why a response ended. A length stop means the response hit the output-token limit.

Take a shell call that runs a targeted test and then prints a short summary. The stream stops before the second half arrives. A lenient parser might still recover a valid object from the first half. The parse says nothing about whether the model finished its command.

If the harness runs that prefix, the model believes it sent one command while the environment ran another. A read then returns an incomplete picture. A write may leave a partial change that no turn intended.

Pi's v0.80.6 loop makes one concrete choice here. If a response ends with a length stop and contains tool calls, Pi creates failed results for those calls instead of running them. The same code validates prepared tool arguments before it runs a call. Arguments that pass validation still do not run if the response was cut off.

A new harness can follow the same rule. Stream to show progress, but run tool calls only from a complete response. Mark an action completed only after it has run.

Carefully designed streaming tools could run each read as soon as it is complete. Such tools need an exact rule for when a call is complete, and a recovery plan for the rest of the response. Build them only for a measured need, not because the transport streams.

The core agent loop fits in a few lines

The pseudocode below shows the loop:

while task is active:
    check run limits
    response = infer(current observations)
    if response is incomplete:
        record failure; recover or stop
    elif response proposes actions:
        execute accepted actions
        append their identified results
    else:
        assess completion evidence
        continue, finish, or report a blocker

In the pseudocode, observations are tool results, actions are tool calls, and completion evidence means records such as the latest test result.

Each line hides a decision, but the loop needs no large framework. In Mini-SWE-Agent v2.4.5, step() queries the model and executes the returned actions. The code around it checks limits, counts consecutive format errors, and saves the trajectory.

For the locale bug, the first tool result might show cache.get(document.id). The second might show that the root test script calls package scripts. The two results justify another turn: find where the locale is normalized, then read the cache tests. No planner object is needed to follow that chain.

A small loop is meant to be easy to debug. If a command runs twice, an engineer should see which branch resent it. If a run ends early, the exit condition should be visible. Line count alone does not make a failure easy to understand. A longer loop can be as clear if it keeps each rule separate.

Mistral Vibe takes another approach. Its v2.19.1 release history records that, in v2.3.0, a middleware pipeline replaced the observer pattern (callbacks that react to loop events) for injecting system messages. Middleware puts small processing steps at defined points in the loop. A step might check the budget before a model call or add a system message before prompt construction. Each step lives outside the core loop rather than as another branch in it.

Our repair might need a budget check before each model call and a context-size check before the prompt is built. Once such checks pile up, middleware gives each one a fixed position and a visible outcome. The cost is ordering. A context-size check that runs before the locale package's newly found instructions join the prompt measures a smaller request than one that runs after. Moving checks into middleware helps only if their order and effects stay visible.

Reads can run in parallel, but edits and tests wait

The agent can often read the cache code and the test configuration as parallel tool calls. Changing the cache key and then testing it must happen in order. The test must see the finished edit. If both run at once, the test can pass on the old code.

Some dependencies are harder to see. One test may fill a temporary cache that another test clears. Two commands that look like reads in the shell can still compete over generated files. Two edits to unrelated files may be safe together, if the edit tool knows which files each one touches.

A cautious first schedule is easy to explain. Run independent reads together. Run changes to the same file one at a time. Finish writes before the tests that depend on them. Adding parallelism should keep these dependencies. The number of tool calls in a response says nothing about safety.

For example, our agent can read the locale code while a package-install check runs. It should not start a second replacement in the preview file, based on the old contents, before the first one finishes.

Result order matters too. If two reads finish in reverse order, each result still needs its own tool-call ID. Otherwise the agent could see the test configuration labeled as the cache code.

Tool errors should say what changed

Assume the edit fails because the old text occurs twice in the file. The tool result should say that no write occurred, that two matches exist, and that the agent needs more context to pick one. A generic “tool failed” leaves the next turn guessing whether the file changed.

Now assume the edit succeeds but the test fails because a package dependency is missing. The code has changed, but the test run has not completed. Replaying the edit would solve neither problem and might damage the correct change.

Tool results do not need a giant schema. Each result needs a clear outcome, a short diagnostic, and its tool-call ID. Tools that change files should also report whether they wrote anything. The harness can store long output separately under a reference the agent can fetch. The model then gets the relevant tail and a truncation marker.

Retries should use that information. Repeating a failed read is usually safe. Repeating a command after a connection failure may be unsafe if the process already started. In this repair, the agent can often rerun an interrupted test after checking its process state. If an interrupted command changed dependencies, inspect the working tree and lockfile first.

Reduce uncertainty before adding another side effect. A retry button that resends the previous string makes the interface convenient and the environment harder to reason about.

Provider adapters translate between model APIs

A provider adapter converts between one model provider's API and the harness's own message format. It translates messages (roles, tool calls, tool results) and metadata (stop reasons, token usage).

Pi's v0.80.6 compatibility types show the details. Its compatibility settings for completion APIs include:

• whether the backend accepts a developer role;
• which maximum-token field it expects;
• whether tool results require a name;
• how reasoning controls are encoded.

Each difference can affect a conversation, even with an endpoint that calls itself compatible.

In the cache repair, the model asks to run a test under a tool-call ID. If the adapter loses that ID, the provider may reject the next request. Or the model may get a result without knowing which call produced it. An adapter can work around a lost ID by turning the tool result into plain assistant text. That may let the next request proceed, but the link between the call and its result is gone.

The harness's internal message format should keep every outcome the application tells apart. If the application treats cancellation and provider failure differently, record them separately. Keep a length stop until the harness has decided whether to run the calls. If usage is unavailable, do not record it as zero cost. You can test each choice without asking a model to solve a repository issue.

A generic adapter can declare which options it supports, instead of quietly accepting a setting it ignores. If the application promises a reasoning budget, an unsupported setting should show up at configuration time or in the run record. Silent approximation makes a benchmark configuration and an interactive session look equivalent when they differ.

Adapters add maintenance. Someone has to follow protocol changes and test the translations. One adapter layer keeps that cost visible. Provider branches scattered through the loop make every tool call depend on hidden provider assumptions.

Budgets and cancellation are part of the loop

A run can look productive and still cost too much or take too long. Our agent might spend dozens of turns on an unrelated package because a broad test command printed distracting failures. Run limits add a stop rule that does not depend on the model.

Each limit needs a defined unit. Model calls, elapsed time, output tokens, and money measure different things. A step budget cannot stop a single shell process that runs forever. A wall-clock limit does not tell you whether the current process was terminated, detached, or merely hidden from the interface.

Mini-SWE-Agent's configuration has separate step, cost, wall-time, and consecutive-format-error limits. A new harness can start with that split. Take the values from your own workload and constraints, not from another product's defaults.

Cancellation stops work in progress when the user or a parent agent asks. If the user stops the run during a file write, the harness cannot assume that aborting the surrounding promise rolls the filesystem back. The harness must wait for or inspect the operation, then report the state it actually found. If cancellation interrupts our test, the result is “verification interrupted,” even if the last displayed line looked encouraging.

The final report can be brief: implementation changed, regression test added, verification interrupted, and these files remain modified. The report describes what actually ran, not what the model said it would do.

How does the agent know the task is done?

The repair has concrete acceptance criteria. Two requests for the same document in different locales must not share a rendered preview. A repeated request in the same locale should still reuse its cache entry. The agent should add a regression test that fails with the old key and passes with the new behavior.

To report success, the loop needs the edit result and a later test result. The agent should also read the final diff, because the working tree began with an unrelated change. The report must say which changed lines came from this run.

Imagine the test passes and the agent then “cleans up” the key construction. A passing test covers only the code as it was when the test ran. The cleanup may be a harmless refactor, but the run has introduced a new claim that needs checking.

A harness can record which files changed after the latest test run, show the actual command, and prompt another check before a success report. The feature must keep the test's scope. A narrow locale-cache test should never become a claim that the full application suite passed.

The final answer should name the changed behavior, the focused test, and any remaining blocker. Without a record of which commands ran and what they printed, a convincing final paragraph is still only a response.

The next article, on how coding agents edit files, follows the edit itself. It covers how a few proposed lines become a patch, why a failed match helps, and what a tool must report when some changes have already landed.

Updated 27 October 2026