When the Agent Becomes the Software

For half a century, software engineering has run on one premise: humans write the decision logic, and the computer executes it. A recent paper by Zhenfeng Cao argues that AI agents are collapsing that arrangement, turning code into something generated and discarded at runtime rather than carefully maintained. The agent itself becomes the software, and the implications reach every team that ships products.

Why software’s complexity ceiling won’t lift

The paper opens with a problem that has haunted the field since Brooks wrote *The Mythical Man-Month* in 1975: large software projects get harder per person as they grow. Brooks split complexity into two kinds. *Accidental complexity* is the friction of a particular implementation, the busywork that better languages, frameworks, and CI/CD (continuous integration and delivery) pipelines can shave away. *Essential complexity* is inherent to the problem itself, and no tooling touches it.

Cao sharpens this with a scaling argument. For a system with n components, each potentially interacting with any other, the number of possible interaction topologies grows as Θ(2^(n²)). Meanwhile, the human capacity to reason about those interactions is, in his words, “essentially constant.” A fixed reasoning budget pressed against an interaction space that doubles faster than you can count. Hierarchical decomposition, the standard engineering response, helps, but as Cao puts it, it “reduces the constant factor but does not change the asymptotic behavior.” That mismatch is the structural reason large projects see declining marginal productivity, and it is the crack the agentic paradigm pushes open.

The shift from code as product to instrument

The paper’s central distinction is formal. A traditional software system is a tuple S = (C, D, E): computational resources C, a set of decision rules D written by humans, and an execution environment E. The defining property is that D is static. Every branch, every edge case, must be encoded before the system ever sees input.

An agent system flips this. Defined as A = (M, T, M, Π), it has an LLM (Large Language Model, a neural network trained on huge text corpora that serves as the reasoning engine) at its core, a set of executable tools T, a memory subsystem, and a planning mechanism Π. The decision logic is generated at runtime. The LLM writes code to solve a step, runs it, and throws it away. Code becomes scaffolding the agent erects and tears down for each task, not the building itself.

This extends Andrej Karpathy’s “Software 2.0” framing, where learned weights replaced hand-written logic. Agents go a step further: the model doesn’t just replace the program, it writes programs on demand. Two techniques make this reliable in practice: ReAct (a framework that interleaves explicit reasoning with tool use) and Chain-of-Thought prompting (asking the model to spell out its intermediate steps), both of which unlock stronger problem-solving than asking for a direct answer. That same decoupling, Cao argues, is what lets agents absorb the complexity that broke the old model.

Three eras, each offloading a new burden

Cao frames commercial software history as a progressive transfer of complexity away from the end-user. Software 1.0 shipped code and data on-premise, sold by license; the customer owned installation and maintenance. Software 2.0, better known as SaaS (Software as a Service, cloud-hosted software sold by subscription), moved code and data into the cloud; the vendor absorbed infrastructure and updates. Software 3.0, which Cao calls AaaS (Agent-as-a-Service, agents autonomously operating in the cloud and priced per outcome), puts the agent in charge of understanding, building, and running.

The qualitative step is in what gets offloaded. SaaS liberated businesses from server rooms. AaaS promises to liberate them from specifying *how* a result should be produced. The customer names the outcome; the agent figures out the rest.

“the agent itself is the software, and its decision logic is generated at runtime.”

— Zhenfeng Cao, *Agentic Software* (Abstract)

Where agents already outperform solo engineers

The empirical record is where the thesis gets concrete. On SWE-bench Verified (a benchmark of real GitHub issues that tests whether an AI can autonomously diagnose and fix bugs in open-source codebases), Lingma SWE-GPT 72B resolves 30.20% of issues, close to GPT-4o’s 31.80%, while being fully open. Its smaller 7B variant resolves 18.20%, a 22.76% relative improvement over Llama 3.1 405B, a model roughly six times larger. The takeaway: training on process data, not just finished code, lets small models punch far above their weight.

A LangChain pilot deploying coordinated agent swarms across 20+ enterprise debugging workflows cut root-cause identification time by 93% and saved over 200 engineering hours in a single month. Critically, the paper notes the gains came “not from better individual agents but from orchestration,” through shared context, parallel investigation, and cross-validation. Hermes Agent, an open-source framework with over 179,000 GitHub stars, demonstrates a self-evolution loop: it writes reusable “Skills” that patch themselves when found lacking, with cross-session memory via FTS5 (SQLite’s full-text search engine).

Two honest caveats belong here. The LangChain numbers come from a blog post, not a peer-reviewed study, with no described methodology or controls. And GitHub stars measure popularity, not validated capability. The evidence is suggestive, not settled.

The cliff agents fall off in real codebases

The same paper that celebrates those breakthroughs reports the most sobering datum. EvoClaw (a benchmark testing continuous software evolution, sustained development across commit histories where errors accumulate) pits agents against the long-horizon work that real maintenance demands. The result: success rates collapse from over 80% on isolated tasks to at most 38% in continuous settings, a 54% drop measured across 12 frontier models in 4 agent frameworks.

“Overall performance scores drop significantly from > 80% on isolated tasks to at most 38% in continuous settings, exposing agents’ profound struggle with long-term maintenance and error propagation.”

— Deng et al., EvoClaw (as quoted in Cao, Section 5.2)

Four named challenges explain the cliff. Context drift: as codebases exceed the model’s context window (the amount of text an LLM can hold in mind at once), agents lose sight of system-wide invariants. Error propagation: a small mistake in an early commit cascades. Technical debt awareness: agents optimize for the immediate task, not maintainability. Verification fidelity: agents can pass tests while introducing subtle semantic errors. Cao’s own calibration is worth keeping: agentic engineering is “real and transformative today as an augmentation paradigm,” but fully autonomous development needs “several more years of concentrated research.”

The engineer’s new job: intent, not keystrokes

If code generation gets commoditized, value migrates. Cao names four new differentiators: intent articulation (specifying goals clearly enough that agents run without producing unintended outcomes), architectural oversight (knowing how multiple agents should coordinate and where human judgment must intervene), quality calibration (defining what “good” looks like and building evaluation frameworks agents can self-correct against), and ethical governance. The paper’s name for the emerging role is “intent architect.”

Software is splitting into two kinds: pre-written logic maintained by humans, and runtime-generated logic produced by reasoning agents. The strategic question for any team is no longer whether to adopt agents, but which of their workflows are simple enough for agents to handle reliably today, and which still need a human in the driver’s seat.