A Firm That Can Survive Its Own Mistakes

Consider a small business that delivers a digital product. Its software accepts orders, collects payment, produces the results, and answers customers. For several weeks, nobody needs to touch it. Then a payment provider changes its interface, and a request times out after a customer has been charged. The business retries, leaving two charges, one delivery, and a customer asking for an explanation.

The business is hypothetical, but the problem gives us a useful test. Someone—or something—has to notice the error, establish what happened, refund the money, and prevent the repair from creating another problem. If the owner has to do that work, the business still depends on a human operator.

The interesting future is an organization that can handle this failure while its owner is absent. For a narrow AI-operated firm, the decisive test should be whether it can recover from mistakes within agreed limits while continuing to meet its obligations. Producing work on a good day gives us a reason to try; surviving a bad day would give us stronger evidence of independence.

The full path from routine automation to removing human oversight, physical intervention, legal signatories, ownership, and ultimately human-originated purpose remains untested. A first experiment can be smaller: one human owner, a business serving human wants, and an operating loop that no longer relies on that person. Judging it requires us to decide which human contributions it claims to remove.

Count the people before counting the agents

An owner who sets the purpose, commits capital, and retains a right to close the business is still a human role. So are customers, regulators, and the people who operate its suppliers. Removing employees does not remove all human dependence. Consider the following three boundaries that make the claim more precise.

Under the first boundary, only people the firm employs or directs count against its claim to be human-free. Under the second, any person intervening in this firm's particular work counts, including a contractor rescuing a failed transaction; people who set general rules in advance do not. The third counts all work done specifically for the firm, even outside it, whether or not it takes the form of an intervention.

Consider the refund again. A business with no payroll but a founder handling billing errors every evening has removed employees while retaining its operator. Buying occasional human repair moves that operator outside the company. This may be commercially sensible, but it does not pass the second test: someone still intervenes in the firm's particular work.

The proposal targets that second boundary within a limited operating domain. It retains human ownership and purpose and assumes shared infrastructure remains available. The stricter boundary and the later steps toward no human role anywhere remain open. Recording every rescue would distinguish progress from a quieter transfer of work to other people.

Better capability changes the problem

With the target defined, we can ask whether improving AI capability gives us a reason to attempt it. METR's March 2025 study found a rising trend in the length of tasks agents could complete at 50% success, measured by how long those tasks took humans. Its methodology warns that the tasks are concentrated in software, machine learning, and cybersecurity and are cleaner than much real work. The result supports testing improving systems, but it is neither a business uptime guarantee nor a forecast of when owners can disappear. METR study, methodology and limitations.

Business experiments show what this progress still leaves unresolved. Anthropic's June 2025 Project Vend report described an AI shopkeeper that lost money and relied on people for physical work. Its December follow-up reported better performance after changes to models, tools, and procedures, while still requiring human purchase checks and substantial support. That improvement weakens the argument that the first failure established a permanent limit, but the remaining support leaves independent operation unproven. Phase one, phase two.

Together, these findings justify testing designs that remove particular human interventions, then measuring which interventions return when something breaks. For our hypothetical business, the next question is what would let it resolve the duplicate charge safely.

Give the firm less authority than intelligence

Resolving that charge requires a trustworthy record, a check against the payment provider's actual state, and permission to issue a bounded refund. It should not require unrestricted access to the company's bank balance or the ability to rewrite spending limits. The agent needs authority to meet the customer's claim, within limits that hold when its reasoning goes wrong.

I would therefore organize the business around commitments it can verify. An order creates an obligation to deliver or refund, so the firm needs to record what was promised and when it is due. A payment identifier lets it reconcile the charge with the provider's record; acceptance criteria let it check the delivery. The agent can choose how to fulfill these commitments, but the surrounding system determines its authority to act.

For this arrangement to work, the records must be preserved separately from the agent's account of what it remembers. Its permitted actions also need limits: how much it can spend, refund, change, or promise in one transaction and across the business as a whole. After an action, it must check the result against outside evidence—a payment record, delivery receipt, or customer report—rather than rely on its own declaration of success.

Those checks will sometimes reveal that the system cannot establish what happened. It then needs to stop accepting affected work while preserving the information needed for repair. Returning to service would require the repair to pass its checks and outstanding customer obligations to be reconciled.

This process would still have to be tested: limits can be specified badly, and external records can be wrong. A correct refund cannot undo every kind of harm. Some failures should end the experiment, making containment essential before any attempt at repair.

The history of automation offers a caution here. According to the SEC's findings, Knight Capital's 2012 trading incident produced more than $460 million in losses in roughly 45 minutes. The SEC identified inadequate controls over outgoing orders and aggregate financial exposure. This was a trading-system failure, not an LLM experiment, but it shows how quickly automated action can outrun effective containment. This is a reason to enforce limits outside the process doing the work, before losses compound. SEC findings.

Stopping is easier than recovering

Containment alone would not resolve the customer's problem. A system that always stops when uncertain can avoid some losses by ceasing to operate, while purchases, refunds, and explanations remain owed. If the owner has to interpret each stop, the business has given that person an alert queue and brought the operator back into the loop.

Recovery therefore needs its own test. Can the business isolate the failed payment integration, preserve its order queue, determine which charges completed, apply a permitted remedy, and reopen without improvising new authority? If it cannot, the missing human judgment tells us what the next design must address.

A reviewing agent might help check the repair, but agreement means little if both agents share the same mistaken assumptions. The reviewer needs evidence that can contradict the actor: a ledger balance, a delivery receipt, a failing acceptance test, or a customer saying that the promised result never arrived. Even different models can share blind spots, so adding reviewers does not remove the need for those checks.

The harder question is whether the rules can remain useful when the environment changes. If unfamiliar events repeatedly require the owner to redefine the problem, ownership has become an operating job again. The system may still save labor, but the stronger claim has failed in that domain. A trial should be designed to reveal that dependence, rather than give the firm only problems its rules already cover.

An experiment small enough to lose

I would begin this trial with one digital service whose output can be checked, whose mistakes can usually be compensated, and whose total exposure can be capped. This is an assumption about where a workable design is most likely to emerge. Success there would say little about a factory, a care provider, or a firm facing open-ended legal disputes.

Before the trial begins, specify the service promise, permitted actions, maximum loss, acceptable outage duration, and conditions for termination. Its accounts should also include compute, infrastructure, refunds, and the human time spent setting up and maintaining the system. Otherwise, an apparently profitable operating loop could conceal expensive human preparation.

Then expose the business to normal demand and controlled disruptions: ambiguous requests, interrupted payments, unavailable tools, defective releases, and attempts to obtain actions outside its mandate. Simulations should come first, followed by real customers only within the agreed limits. A human must remain able to stop a dangerous trial; exercising that power counts as an intervention and belongs in the results.

Those results should show fulfilled obligations, unreconciled money, downtime, repeat failures, total cost, and every minute of human rescue. Compare them with a human-operated service under similar demand and exposure to test whether independence offers a useful gain. Automation need not be perfect, but it does need to meet the service promise and stay within limits chosen before seeing its performance.

Several findings would defeat the proposal for the tested business: recurring owner interpretation despite successive repairs, losses above the stated ceiling, shared errors that defeat both production and review, or a cost of independence that consistently exceeds the value delivered.

A clean trial would support only that design, domain, and tested range of conditions. It would not establish that human ownership, legal responsibility, physical repair, or purpose can be removed. A failed trial would identify a dependency to investigate, rather than prove that every future design must fail.

What changes if the test starts passing?

If trials begin to pass, a person could own a useful service without becoming its permanent dispatcher. Keeping promises while the owner is absent would leave human effort directed toward choosing purposes, financing experiments, and deciding which risks are worth taking.

If that arrangement spread through a sector, competitors could respond by reducing prices, expanding service, or starting more businesses. Yet those responses would depend on access to customers, infrastructure, and capital. Where many firms relied on a few providers, those providers could capture much of the benefit. Removing operating labor alone would not guarantee open competition.

Widespread adoption would expand the test from whether one firm can recover to whether many can survive a shared failure. A common provider outage or mistaken update could disable organizations that looked independent in isolation. At that scale, greater productive capacity would still leave open who owns it and how people obtain the income to buy its output. These are conditional consequences, beyond what the proposed trial could demonstrate.

The next firm worth building may have one owner and no routine operator, but removing the need for a human rescue cannot remove the customer's claim on the business. The duplicate charge gives us a concrete place to start: which failures could this firm repair today, and what would have to change for it to repair the next one?

Sources

METR studymetr.org
methodology and limitationsmetr.org
Phase oneanthropic.com
phase twoanthropic.com
SEC findingssec.gov