Autonomous Improvement Rate - Technical Notes

← Read the introductory version

The introductory version of the Autonomous Improvement Rate is intentionally simple. These notes make the assumptions more explicit: what the terms mean, how they interact, which parts generalize across domains, how the model can fail, and what observations would count against it.

What the equation describes

AIR models a closed improvement loop. A system takes an action or runs an experiment, observes the result, evaluates what it means, retains useful evidence, and lets that evidence influence what it does next. The model asks how quickly that loop produces measurable progress against a declared objective.

“Iteration” is deliberately broad. In software it can be an implementation/build/test cycle. In simulation it can be a model run. In evaluation infrastructure it can be a held-out experiment. In an automated laboratory it can be a physical experiment. The mechanics differ, but each case has a throughput, an information yield, and some ability to carry what was learned into the next decision.

Dimensional statement

The relation is proportional rather than a literal universal equality. A useful dimensional interpretation is:

• E_t - iterations · time⁻¹
• I_E - useful information · iteration⁻¹
• η_L - Δperformance · useful-information⁻¹

Their product has units of Δperformance · time⁻¹, which is what an improvement rate requires. E_t × I_E is an effective information-acquisition rate; η_L describes how effectively that information becomes progress against the objective.

The objective has to be named before the model is measurable: test pass rate, held-out performance, yield, time-to-completion, or another criterion declared before the search begins. Without an objective, “improvement” has no direction.

The three terms do not generalize equally

A useful architectural consequence of AIR is that the three factors have different degrees of domain dependence.

• E_t is largely a systems problem. Execution engines, parallelism, scheduling, caching, retries, containerization, and orchestration can often be reused across many kinds of computational work.
• η_L is largely a learning and memory problem. Persisting findings, retrieving prior evidence, recording decisions, and making previous results available to later agents or iterations are also substantially reusable patterns.
• I_E is much more domain-specific. What counts as new evidence, whether it is trustworthy, whether the evaluator is a faithful proxy for the real objective, and whether a result should change the next decision depend heavily on the problem being solved.

None of the terms is completely domain-independent. But there is a meaningful reuse asymmetry: the machinery that runs iterations and preserves learning can often be generalized farther than the machinery that decides what the results mean.

That suggests a practical architecture for autonomous research and engineering systems: build reusable infrastructure around throughput and learning retention, then plug in a domain-specific evidence and evaluation layer.

The factors are coupled

The simple form above is a useful first-order model because it treats throughput, information yield, and learning efficiency as separable factors. That makes the dimensions and engineering roles easy to see. But the factors are not actually independent in a working system.

The most important coupling is between iteration throughput and information per iteration. Many of the mechanisms used to increase E_t - batching, concurrency, speculative execution, and parallel experiments - also change what each iteration can learn. Ten sequential experiments can make experiment ten depend on what experiments one through nine discovered. Ten experiments launched simultaneously cannot. Higher throughput can therefore produce more observations per unit time while making each observation less informative on average.

To make that dependence explicit, the first-order model can be refined by treating information yield as a function of throughput:

This is not a different theory; it is the original decomposition with one independence assumption relaxed. I_E(E_t) means “expected useful information per iteration at a given iteration throughput.” The subscript t in E_t is a mnemonic for throughput, not a time index, so it does not imply E at time t or a dependence on E_t−1.

A batch of parallel experiments can therefore contain redundant probes that a sequential process would have skipped after seeing earlier results. Parallelism can still be the correct trade - especially when iterations are cheap - but its additional throughput may arrive with an information discount.

The engineering objective is not to maximize E_t, I_E, or η_L independently. It is to find the operating point that maximizes their product under the system's real constraints.

Learning efficiency has two parts

η_L is easier to reason about if it is split into two mechanisms:

• Retention: Does useful evidence survive into later iterations, sessions, and agents?
• Exploitation: Does the retained evidence actually change a later action?

A system that repeatedly rediscovers the same fact has a retention problem. A system that remembers the fact but keeps making the same choice has an exploitation problem. Both reduce effective learning efficiency.

Information quality and verification fidelity

I_E cannot simply mean “bits produced.” An experiment can generate enormous amounts of data without teaching the system anything useful. For AIR, the relevant quantity is information that reduces uncertainty or improves a decision relative to the declared objective.

This is where domain knowledge enters most strongly. A compiler and test suite can provide strong feedback for some software tasks. A scientific model needs measurements that discriminate among hypotheses. An evaluation pipeline needs evidence that survives out-of-sample testing. A robot ultimately needs evaluation against the physical world rather than only against a simulator.

Verification fidelity determines whether that information points in the right direction. If the evaluator rewards the wrong thing - a weak test, misleading simulation, overfit evaluation, or other corrupted proxy - the system can adapt efficiently to the wrong objective. In that operational sense, effective η_L can become negative: more throughput and faster adaptation make the system worse faster.

This is why the domain-specific evaluation layer is not an accessory to autonomous improvement. It is what turns activity into trustworthy evidence.

A cost-per-iteration prediction

The decomposition suggests a checkable prediction: as the cost of an iteration rises, the optimal allocation should shift from iteration throughput toward information gained per iteration.

Cheap, reversible domains can afford many attempts. Expensive physical experiments have stronger incentives to choose each experiment carefully. If iteration cost falls sharply in a domain, the optimal balance should move toward more exploration and higher throughput; if iteration cost rises, experimental design should become more valuable.

The constrained form

Real systems optimize under budgets. The useful question becomes: maximize R_A subject to limits on money, compute, elapsed time, human attention, physical materials, or access to scarce models.

In my current software-development work, local inference is comparatively abundant while frontier-model access is constrained. That makes Autonomous Completed Work per Frontier-Model Token a useful operating metric. It asks whether improvements to orchestration are converting scarce high-capability inference into more completed work rather than merely producing more agent activity.

Instrumentation

The exact measurements must vary by domain, especially for I_E. These are engineering proxies rather than universal definitions:

Concrete examples from my current systems

In my software-development workflow, Ratchetloop is the bounded delivery loop: implementation in isolated worktrees, deterministic checks, independent review, persistent run evidence, and human promotion. It attacks E_t by making verified software iterations faster and less dependent on continuous human supervision.

Its branch, event log, run record, review findings, and repository decision records attack part of η_L: useful evidence should survive the worker or context window that produced it and remain available to later work.

The information-quality layer is necessarily more specific to the problem. Strategy Assayer is one example of evaluation infrastructure. It separates search fitness from walk-forward and out-of-sample evidence so higher search throughput does not simply produce faster overfitting. A chemistry lab, robotics system, or other research domain would need its own equivalent machinery for deciding what constitutes credible evidence.

These systems are not presented as a complete implementation of AIR. They are concrete examples of the three engineering problems the framework separates: run the loop efficiently, determine what the results actually mean, and make what was learned available to the next loop.

From software loops to computational research

Software is a particularly fast proving ground because iterations are cheap, feedback can often be automated, failures are usually reversible, and many experiments can run in parallel. The same structure extends naturally to simulation and computational research: propose an experiment, execute a model, measure the result, evaluate the evidence, retain what was learned, and choose the next experiment.

I use Autonomous Computational Research as a descriptive term for systems that automate substantial portions of that computational research loop - not merely executing a researcher-supplied batch of jobs, but helping select, run, evaluate, and learn from successive experiments under explicit objectives and constraints.

AIR is intended as a way to reason about the improvement rate of such systems, but it is not limited to them. The same decomposition can be applied wherever autonomous work has an iterative action/evaluation/learning loop, including automated laboratories, robotics, manufacturing, and other physical systems.

Flywheel dynamics are a separate question

R_A describes an improvement rate at a point in time. It does not, by itself, prove that improvement will accelerate indefinitely. The next useful improvement may be harder to discover than the previous one, and diminishing returns can overwhelm positive feedback.

The recursive loop - better system → more autonomous work → faster system improvement → better system - is therefore a mechanism for feedback, not proof of runaway growth. Whether the loop accelerates, stabilizes, or slows is an empirical question.

What would count against the framework?

• Systems with comparable E_t, I_E, and η_L persistently improving at very different rates without another explanatory factor.
• Large changes in iteration cost producing no corresponding change in the optimal balance between throughput and experiment quality.
• Measured improvement persistently outrunning the fidelity of the evaluation signal used to detect it.
• Domains in which the proposed reuse asymmetry consistently fails - for example, where throughput and retention infrastructure prove just as domain-specific as the evidence layer.

The first and fourth cases would suggest the decomposition or its architectural interpretation is missing something important. The other cases would challenge predictions derived from the framework rather than the dimensional decomposition itself.

Relationship to earlier work

None of the individual ideas starts here. Iterative improvement cycles long predate modern AI. Expected information gain comes from Bayesian experimental design and active learning. Throughput is a basic systems and operations concept. Knowledge retention and exploitation are central problems in learning systems and organizations.

The narrower claim is the multiplicative rate model and its generalization across autonomous domains: three named factors whose product provides a useful model of improvement rate, together with the engineering consequences that follow from their coupling, different degrees of domain dependence, verification fidelity, and resource constraints.

Closest prior art

A closer antecedent is Volk & Abolhasani, Performance metrics to unleash the power of self-driving labs in chemistry and materials science, Nature Communications 15, 1378 (2024), DOI: 10.1038/s41467-024-45569-5. Their performance framework explicitly separates experimental throughput from optimization performance and also treats precision, sampling cost, accessible parameter space, operational lifetime, and degree of autonomy as distinct system characteristics. Importantly, they argue that optimization rate alone is not sufficient for comparing self-driving laboratories. This is closer to AIR's core decomposition than a later review because it treats the rate of experimentation and the quality or efficiency of what the search extracts from those experiments as separate quantities, even though it does not propose AIR's multiplicative rate equation or generalize the framework beyond self-driving laboratories.

Lee et al., Toward self-driving laboratory 2.0 for chemistry and materials discovery, Materials Horizons (2026), DOI: 10.1039/D5MH01984B, is better understood here as a later synthesis. That review describes self-driving laboratories as advancing along three interdependent axes: experiment throughput, information gained per experiment, and efficiency of converting data into scientific knowledge. AIR generalizes and formalizes this family of SDL performance ideas into a domain-independent multiplicative rate model, then develops consequences of factor coupling, verification fidelity, resource constraints, and application beyond physical laboratories to software and computational research systems.

Todd Espy · August 27, 2026 · Technical notes

Generated output stats available on request.