Self-Improving Agents: From One-Shot Prompts to Closed Reasoning Loops
How coding agents go from one-shot code generators to systems that check and fix their own work, using test-time compute and real compiler and test output.
The first wave of AI code generation treated programming as text completion. You wrote a prompt. The model wrote 200 lines of code in one pass. Then you spent the afternoon debugging its hallucinations (code the model made up, like functions that do not exist).
One-pass generation is fine for boilerplate. It breaks on real engineering work: refactors across many files, keeping asynchronous state consistent, and dependency migrations.
Agents got reliable when they moved from single-pass prompting to closed reasoning loops. In a closed loop, the agent writes code, checks it, and uses the result to choose its next edit. Research backs this up, starting with the Self-Refine paper (Madaan et al., arXiv:2303.17651) and agent designs like Nous Hermes.
The Flaw of Single-Pass Generation
An LLM writes one token (a small chunk of text) at a time. Each token depends on the ones before it. This is called autoregressive generation, and it has no undo. Say the model makes one small wrong assumption at token 40, like an outdated database schema or a deprecated method signature. Every token after that has to work around the mistake. The model does not fix the error. It builds on it.
On hard tasks, this causes three failure modes:
- Compounding Syntax Errors: One missing bracket or bad import causes dozens of compile errors further down.
- Hallucinated State: The model has no sandbox to run code in. So it guesses what terminal commands and database queries would return.
- Premature Convergence: The model commits to the first design that looks plausible. It keeps that design even after later constraints make it unworkable.
The Self-Refine Framework (arXiv:2303.17651)
In Self-Refine: Iterative Refinement with Self-Feedback, Madaan et al. ran LLMs through a repeated three-step cycle. Performance went up by as much as ~20%:
- Generate: Write a first version.
- Feedback: Check that output against the requirements and edge cases.
- Refine: Change the output, guided by that feedback.
Here is the important part. The models did not need retraining: no supervised fine-tuning (training on labelled examples) and no reinforcement learning (training on reward signals). They improved at test time, while answering. This is test-time compute: compute you spend while the model works on a task, not while it trains. Here, that means more passes over the same problem.
Grounded Feedback vs. Sycophantic Self-Critique
In the original paper, the model wrote its own critiques. For code, that runs straight into LLM sycophancy: the model’s habit of saying what sounds agreeable instead of what is true.
Ask a model to review its own code with nothing to run, and you get confirmation bias. It either invents flaws that do not exist or calls broken code correct, with full confidence. Neither helps you.
So in a coding agent, self-refinement must be environment-grounded. The feedback has to come from running the code, not from the model’s opinion of it:
| Feedback Mechanism | Grounding Source | Failure Mode |
|---|---|---|
| Prompt Self-Reflection | Pure LLM token probability | Confirmation bias, sycophancy, missed runtime errors |
| Compiler Feedback | tsc, rustc, gcc exit codes | Syntax-only (does not verify semantic logic) |
| Test-Driven Refinement | Unit test assertions, linters, MicroVM runtimes | Checks real behavior; same result on every run |
Replace text critiques with compiler errors and test output, and the loop gets ground truth: a signal that reflects what the code actually does. A failing test does not flatter you.
Closed Reasoning Loop Architecture
An autonomous agent runs this cycle on its own, with no human in the middle:
Generate
Make one scoped file edit, using AST symbol context.
Diagnose
Compiler and tests report exit codes, typecheck traces, and lint errors.
Fix the failure
The error output goes into context. The agent writes a targeted diff.
Git checkpoint
All tests exit 0. The agent commits and waits for the next instruction.
The Hermes Pattern: Structured Agentic Scratchpads
Nous Research builds the Hermes line of agent models. Hermes keeps two things strictly apart: private scratchpad reasoning (notes the model writes to itself) and public tool invocation (the calls it actually makes):
- Explicit Pre-Action CoT: CoT is chain of thought, the model’s step-by-step reasoning written out as text. Before every file edit, the agent writes down why the edit is needed.
- Single-Concern Invocation: Each tool call does one thing. A call that does five things hides which one broke.
- Structured Diagnostics: The runtime returns stdout, stderr, and exit codes as strict JSON, not raw text.
When a test fails, the agent does not start over. It keeps its verified scratchpad history. It finds the one assumption that was wrong. Then it writes a small patch for that assumption only.
How December Implements Self-Improving Loops
December builds the Self-Refine loop into its workflow, from the terminal to the cloud:
- Isolated MicroVM Verification: Every terminal action runs in a sandboxed Linux container. Tests and builds run there in real time.
- Verifiable Git Checkpoints: Before December changes any file, it snapshots the working tree. A refinement loop can drift, or fail to converge (never reach passing tests). If that happens, run
/undo. It restores the tree to the exact state before the run. - Pruned Diagnostic Memory: December does not paste thousands of lines of terminal logs into the prompt. It parses compiler stderr into short diff hints. This keeps the prompt small and avoids context rot (the drop in accuracy as a prompt gets longer).
Point test-time compute at real compiler and test output, and a coding agent stops guessing. It becomes an engineer that checks its own work.
References: