The risk with agents is no longer only what they write, but what they can do
Development agents are crossing an important threshold: they no longer merely suggest code, they execute work. They inspect repositories, edit multiple files, run tests, consume compute credits, open pull requests, and can chain those actions during long sessions. For a software team, that is a real promise of speed. It is also a change in kind: the issue is no longer only the quality of a generated snippet, but the control of a system that can act.
Two recent publications point in the same direction. Gartner argues that agentic AI governance cannot remain a set of written policies: it has to become a set of runtime controls able to stop a risky action before it touches production, credentials, budgets, or data. A synthesis paper on arXiv describes the move from simple code generation to an agentic software development lifecycle where value is measured in changes that are actually qualified for production, not in the volume of lines produced.
The practical conclusion is simple: humans must remain in command, but not as heroic validators placed only at the end. They must define boundaries, choose autonomy levels, require traces, organize review, set budgets, and decide which actions need explicit approval. In other words, the human role shifts: less micro-control of every suggestion, more control of a delivery system.
Why paper policies are no longer enough
In a traditional process, a security or compliance rule often applies in stages: team convention, pull request checklist, human review, CI pipeline, validation before deployment. That model works because actions move at human speed and because the gates are relatively stable. An autonomous agent changes that assumption. It can perform dozens of operations in minutes, combine tools, retry commands after failure, and revise its plan according to observed results.
If the only protection is a policy in an internal document, it arrives too late. It describes what should have happened, but it does not prevent a destructive command, an information leak inside a prompt, excessive token consumption, or a pull request that mixes a useful fix with hidden debt. Governance therefore has to move from commentary to execution.
For a development team, this means treating every agent as a governed identity. Who launched it? On which repository? With which rights? For which task? For how long? Does it have access to the network, secrets, local databases, customer tickets, or staging environments? These questions sound like IAM, DevOps, and application security, but they also become product-engineering questions. An agent that can modify code is an actor inside the software system.
The control plane for a software team
An agentic control plane does not have to start as a complicated product. It is first an architecture of decisions. Teams can begin by classifying actions by risk. Reading public code, generating a unit test, or proposing documentation can be allowed broadly. Editing a database migration, changing an authentication policy, touching cloud configuration, or running a destructive script should require explicit human approval.
The second building block is the execution environment. Agents should work in isolated branches, local or remote sandboxes, and with minimal temporary access. Secrets should not be available by default. Network access should be limited when the task does not require it. Sensitive commands should be blocked or turned into approval requests. This is not distrust of the tool; it is the same logic that led teams to separate development, test, and production.
The third building block is traceability. A pull request produced with agent help should tell the story of what happened: initial intent, modified files, tests run, failures encountered, assumptions made, and areas not verified. Without that log, the human reviewer faces an opaque final result. With it, the reviewer can focus on the important decisions instead of reconstructing the path.
- Identity: every agent has a human owner, a scope, and limited rights.
- Tiered autonomy: reversible actions can be automated, irreversible actions remain approved.
- Sandbox: the agent works in a controlled space, not across the entire machine or cloud.
- Budgets: tokens, CPU time, tool calls, and CI executions are capped.
- Observability: prompts, decisions, commands, and critical outcomes are recorded.
The real metric: a production-qualified change
The trap with coding agents is to measure what they easily produce: number of files changed, generation speed, amount of tests added, volume of pull requests opened. Those indicators can create the impression of progress while shifting cost to review, integration, security, or operations. The arXiv paper describes exactly this throughput paradox: the agent can accelerate code production faster than the organization accelerates its ability to qualify that code.
A better metric is the production-qualified change. A qualified change is not merely code that compiles. It satisfies the business intent, respects the architecture, passes relevant tests, does not break existing contracts, does not expose sensitive data, remains observable in production, and can be explained by a responsible human. This definition puts the human in the right place: not as an obstacle to speed, but as the guarantor of delivered value.
In this model, AI is extremely useful. It can prepare alternatives, write tests, summarize impact, look for possible regressions, generate migration documentation, and propose a rollback strategy. But the movement from plausible to acceptable remains an engineering decision. A model can help formulate the argument; it does not carry responsibility for the service in production.
What teams can do now
The first action is to make red zones explicit. Many teams implicitly allow agents inside real repositories without defining what they must never do. Teams should list dangerous commands, sensitive files, services that must not be called, data that must never enter a prompt, and types of change that require senior review. That list should live with the repository, like CI rules or security conventions.
The second action is to change AI-assisted pull requests. A PR should indicate whether an agent contributed, which instructions it received, which tests it ran, and which parts remain to be checked. This is not about stigmatizing the tool. It helps the reviewer calibrate attention. A quickly generated PR can be excellent; it simply needs review context that matches how it was produced.
The third action is to define budgets. Agents have variable cost: model usage, sandboxing, tools, CI, trace storage, review time, and rework. Without budgets, the team discovers the bill after the fact. With budgets, it can decide which tasks deserve higher autonomy, which should remain manual, and where assistance produces a real net gain.
The fourth action is to train developers to become better operators of agents. That includes writing precise intent, decomposing tasks, checking outputs, reading diffs critically, designing tests as oracles, and taking back control when the agent drifts in the wrong direction. The profession does not disappear; it moves toward responsible orchestration.
Keeping human command without slowing everything down
Keeping humans in command does not mean asking for manual approval on every line of code. That would be inefficient and, paradoxically, dangerous: reviewers would end up clicking mechanically. The better model is autonomy proportional to risk. The more reversible, isolated, and observable an action is, the more it can be automated. The more it touches security, data, cost, or production, the more it must pass through explicit human control.
This approach is compatible with speed. An agent can move quickly inside a narrow scope. It can prepare a change, run tests, document its choices, and propose follow-up actions. The responsible human intervenes where judgment matters: intent, trade-offs, architecture, risk, and final acceptance. That is a better division of labor than the fantasy of a fully autonomous agent or the suspicion that blocks every experiment.
Team maturity will therefore not be measured by the number of agents deployed, but by the quality of the guardrails around them. The organizations that win will not be the ones that remove humans from the loop. They will be the ones that design a loop where the agent accelerates execution and the human keeps the power to say no, demand evidence, and own the final decision.
