Thursday, 9am At nine on Thursday morning a container started in Azure, pulled the week's sources, clustered the same story where several outlets had covered it, ran a novelty check that threw out the duplicates and anything it had already published, capped each topic to a handful, then selected and rewrote 69 articles across four editorial voices, generated the images, wrote everything to the production database and told the host to rebuild the site and refresh the search index. Nobody was watching. I was making coffee. The first human to read any of those articles, was a reader. Then the same job scored its own output. Every article went through mechanical checks, a Haiku judge tested a sample for claims the sources do not support, and a multi judge persona panel scored a stratified dozen. The summary landed in blob storage next to the telemetry: banned phrases 0, structure 1.0, groundedness 0.98, and cost $2.26. If you'd asked me eighteen months ago whether I would let a language model publish to production with no human in the loop, I would have said no, and I would have felt responsible saying it. I still feel responsible but that "no human in the loop" stopped being a decision I make and became a rung the system has to earn. That sentence is not original - "Autonomy is earned, not granted" is on half the vendor blogs of 2026. What you will struggle to find written down though is what the receipt looks like: what evidence, gathered how, over how long, against what bar, and what happens when the evidence goes bad. This piece is my answer, with the numbers from a pipeline that has been climbing the ladder since July. The ladder What does an agent have to show you before you let it act? My answer is a table. Every automated step in my pipeline sits on one of four rungs, the rung is a config value, and the code refuses to act above the rung you have given it. Rung The agent may You do It moves up when 1. Shadow run, and write its output to a log the work, as before its logged output would have been right often enough, measured over several real cycles 2. Advisory propose: a recommendation, a diff, a score decide, every time accept and override rates are healthy and the overrides have a pattern you can name 3. Assisted act, with a check it must pass first approve or veto the result, not the steps the check catches every failure on its list and review time per item is measured and falling 4. Autonomous act, unattended read the scorecard the scorecard has been green for long enough to trust, against a threshold you set before you saw the numbers Three rules make it a ladder rather than a diagram. Promotion needs evidence you gathered on the rung below, over more than one real cycle, against a bar you agreed in advance. Not a demo, not a benchmark, not a week of it looking fine - actual scored cycles of the real job. Demotion is one config change, made the moment the evidence slips. That's the rule we'll all be tempted to skip. An autonomy scheme without a demotion path is a launch plan. The whole point is that trust is a variable you keep measuring, not a milestone you pass once. And the rung belongs to a step, never to "the agent". The rewrite stage of my pipeline is on rung four. The ops-doctor that diagnoses its failures is on rung one. Same pipeline, same model family, same week. An agent is not trustworthy or untrustworthy. A specific action class, with a specific evidence trail, is. Where the rungs come from None of the rungs are mine, and you should be suspicious of anyone who says otherwise. Andrew Ng taught the spectrum in his MLOps course in 2021: human-only, then "shadow mode, where the learning algorithms are giving predictions but are not actually used", then AI assistance, partial automation and full automation, with the advice that "many deployment applications will start from the left and gradually move to the right". Behind Ng sits a forty-year-old scale. Sheridan and Verplank's ten levels of automation from 1978 run from "human does the whole job up to the point of turning it over to the computer to implement" through "computer helps determine options and suggests one, which human need not follow", "computer selects action and implements it if human approves", "computer selects action, informs human in plenty of time to stop it", to "computer does whole job if it decides it should be done, and if so tells human, if it decides he should be told". My advisory rung is their levels two to four. Assisted is five and six. Autonomous with a scorecard is seven: "computer does whole job and necessarily tells human what it did". Shadow mode is the one rung Sheridan does not have, and it came from cars. Tesla has run its driver-assistance software in shadow on customer vehicles since 2016, predicting what it would do and logging the moments its prediction disagrees with the driver. By 2019 the pattern was standard in the machine-learning deployment literature: deploy the new model "side-by-side with the current one, as a shadow model, and send the same production traffic to gather data on how the shadow model performs before promoting it". Note the verb. Promoting. So the ancestry is honest and the pieces are old. What the older frameworks do not have is the bit in the fourth column. Sheridan's scale is a design choice: you pick a level for a function at design time whereas Ng's spectrum has a direction but no gate. The deployment literature promotes models, not authority, and jumps from shadow straight to serving. The question none of them answers is the one OpenAI's 2023 governance paper left open: "How should a user or system deployer determine and enforce which interactions need human approval, as a function of a model's empirical reliability on that task and other factors?" The ladder is my answer to that question. The rest of this piece is what the answer looks like when it is running. Historical Precedents If the ladder sounds like a novelty from the AI industry, look at how many older professions grant authority in exactly these steps, and how many of them wrote the evidence bar down as a number. A junior doctor in a competency-based programme is entrusted with a professional activity on a scale that starts at "not allowed to practice; allowed to observe", moves through supervision "with supervisor in room ready to step in as needed", then "supervisor immediately available, all findings double checked", then only "key findings double checked", and ends with practising unsupervised. Observe, co-act, act-with-check, act. The Ottawa version of the same scale is written from the supervisor's side and reads like a changelog of trust: "I had to do", "I had to talk them through", "I had to prompt them from time to time", "I needed to be in the room just in case", "I did not need to be there." Aviation is stricter and more numeric. When an airline wants approval to let the autoland system put an aircraft on the runway in fog, the FAA's classic criteria asks for "at least one-hundred (100) successful landings" in line operations first, with data collected on every attempt "regardless of whether the approach is abandoned, unsatisfactory, or is concluded successfully", typically over 6 months. Authorised minima then step down in stages, from 300 metres of visibility to 175 to 75, each step earned by the record of the one before. If "an excessive number of failures" turn up during the demonstration, the answer is more landings or a system modification, not a lower bar. The bank regulators got there in 2011. The Federal Reserve's model risk guidance says that if validation finds significant deficiencies, "use of the model should not be allowed or should be permitted only under very tight constraints until those issues are resolved", and it describes shadow mode almost exactly: "parallel outcomes analysis, under which both the original and adjusted models' forecasts are tested against realized outcomes", where "if the adjusted model does not outperform the original model" it does not replace it. It even asks for "early warning" metrics that start shortly after model introduction. That is the whole ladder, in supervisory prose, fifteen years before anyone put an LLM in a loop. And the deployment engineers have had the mechanics for years. Google and Netflix's canary analysis tool routes a small slice of traffic, one per cent in Google's example, through the change, scores it against a baseline, and then, in a sentence that contains all four of my rungs: "Success promotes the canary and continues the deployment, a marginal score can trigger a human approval path and failure triggers a roll back." The success and failure conditions live in a template, written before the rollout starts. So neither the pattern nor receipts are new. What's new is that the thing climbing the ladder now writes prose, makes API calls, files pull requests, and could delete your production database and most of us forgot to bring the receipts. The rewrite stage: four rungs in eight weeks The article-rewriting stage is the clearest example because I can show you every step. It started life on a local Qwen model, which had run unattended for a year, and the question in July was whether to move it to hosted Claude. So Claude went into shadow - which is where your new model should always start. For the full weekly set, both backends rewrote the same 85 articles, the outputs were logged, nothing from the new backend was published, and a blind panel of two judges from different model families scored both. Cloud 4.73 out of 5 against local 2.19, every persona, both judges independently agreeing. I wrote that up in the judge panel piece, including the part where the gate could pass on zero judgments. That shadow evidence promoted the cloud backend straight past advisory and assisted, because the job it was replacing was already autonomous, and the promotion question was only "is this at least as good". Then the first production edition ran with me watching, which is rung three in all but name. Then again, unattended. Since then the cadence has moved to three editions a week, Wednesday, Friday and Sunday, with cross-run deduplication against the published corpus enforced from the first Sunday run. Same rung, more cycles, every one of them scored. But autonomy for the rewrite stage came with a condition: online evals on every edition, thresholds fixed before the first run. Persona adherence at least 4.0 mean, groundedness at least 0.95, banned phrases zero, structure 100 per cent. Miss a threshold and the run is marked degraded in the run state and a human is told. The thresholds are a promise I made to myself in a config file before I had any production numbers to be tempted by. Eugene Yan is right that setting a threshold is hard and that being off by 0.05 can swing precision or recall badly, and the honest answer is that mine were calibrated on the shadow data and then frozen. That is the order that matters. Calibrate on evidence from the rung below, freeze, then run. Never the reverse. The gate is on rung one too Here is the part I only noticed while writing this, and the part you are most likely to get wrong. The quality gate I just described is itself a new automated step, and it is on rung one. It is report-only. It can mark a run degraded, it can shout, and it cannot block a publish. Your first instinct will be to let it block. Resist that. It will earn blocking the same way everything else does: by being right for several cycles in a row. Thursday's run showed why. The persona panel scored nothing. Both judges failed, and the summary said so in a field called judges_degraded rather than inventing a number. That is the fix from the judge panel piece holding up in production. But the reasons the judges failed were three separate bugs that had never had real input before: one judge rejected the schema because a dial name contained an ampersand, one had a missing dependency in the container image, and one had the wrong response shape hard-coded from the A/B run. All three are fixed. All three would have blocked a perfectly good edition if the gate had been allowed to block. I believe that a gate that starts life blocking is a gate you will disable the first time it is wrong but a gate that starts life reporting is one you will still have around in six months. The fixed panel, tested on three real articles, scored persona adherence at 3.0 to 4.4 against a threshold of 4.0, and in the first draft of this piece I wrote that the first fully scored run would probably read "below threshold" on that dial and that I would not move the bar. I was wrong about the number and I am keeping the sentence, because the wrong prediction is the useful part. The next scored run, the day after, judged all twelve sampled articles and the panel came in at 4.22. Persona cleared. What did not clear was groundedness: 0.894 against a bar of 0.95, and the run was marked degraded, report-only, exactly as designed. So the gate's first real signal was on the dial I was not watching, about the thing I care about most, and it came from a bar I had set before I had any numbers to be tempted by. That is what the bar is for. Moving the threshold to meet the result is how you get a gate that never fails. Hamel Husain puts the inverse well: if you are passing 100 per cent of your evals, you are probably not challenging your system enough. The ops-doctor: live on rung one, staying there for now The newest agent in the pipeline is an ops-doctor. It went live on Friday, the day after the first scored run, because a scored production run existing was its go-live condition. When a run fails operationally, a stage exits non-zero, an edition comes out empty, a batch is orphaned, the judges cannot start, it reads the failure context, classifies the cause against a small taxonomy (revoked key, source site changed, rate limit, code regression, dependency drift, config, image build, secret rotation) and files a structured diagnosis as a GitLab issue. That is rung one. It may diagnose and report, and you will notice it is not allowed to touch code. It may not propose a patch, it may not open a merge request, and it certainly may not merge one. The rungs above it are designed and written down, and none of them exist as code, which is deliberate. Notice also what does not wake it: a quality flag. A groundedness score of 0.894 is a signal for an editor, not an incident, and the first version of the trigger fired on any degraded run until we narrowed it, because a doctor that files an issue every time the prose scores 3.9 is - you guessed it - a doctor you will learn to ignore! The two signals go to two different places on purpose. Its promotion evidence is a casebook: every failure it has been given with the human-confirmed cause alongside its diagnosis. Three cases I built by breaking a dev copy on purpose. Two came from a real test in August. The other five came from the cloud cutover a week ago, where five infrastructure gaps were stacked one behind the other, each invisible until the one in front of it was fixed. A doctor that had seen only my synthetic cases would have been useless on that afternoon. The golden set for an agent that diagnoses failures has to be built from real failures, and the cutover donated half of it. The entry bar for switching it on was five cases correct. It got ten out of ten, twice. In between it got nine out of ten, and then nine out of ten on a different case, which is the more useful result. The classifier runs at a low temperature but not zero, so two ambiguous boundaries in the taxonomy showed up as flicker: rate-limit versus judge-failure, revoked key versus secret rotation. I tightened the definitions rather than the temperature, because a taxonomy that only works at temperature zero is a taxonomy with a bug in it. Ten for ten on a casebook I partly built by breaking things on purpose is not evidence for rung two, and if you are tempted to read it that way, ask what the casebook does not contain. It is however evidence that it works well enough to be worth running. Rung two - proposing a diff inside the issue - needs at least two real production cycles where its diagnoses are read and marked right or wrong. If the marks come in at 80 per cent or better, it moves up. If not, it stays, and I learn something about the taxonomy. What the vendors already ship - and what they don't I went looking for who else does this, because the ops-doctor is a small copy of a pattern every observability vendor now sells. What I found is that the industry has settled on the rungs and skipped the receipts. Datadog's Bits AI SRE, since renamed Bits Investigation, is "an autonomous AI agent that investigates production issues end to end", and then "engineers can review and merge the resulting pull request". Autonomous investigation, assisted remediation. incident.io is blunter: "the only change Investigations can make to your systems is a pull request you review and merge yourself". Microsoft's Azure SRE Agent is the most interesting, because it publishes a ladder: "Review is the default mode. Your agent investigates, identifies a fix, and asks for your approval before executing" infrastructure changes, and then, "Start with review mode. Observe what the agent recommends for two to four weeks. When you find patterns you consistently approve, switch those specific triggers to Autonomous." Read that last sentence again, because it is nearly right and the gap is the whole subject. The promotion criterion is that you, a person, notice a pattern in your own approvals over two to four weeks. No number. No bar set in advance. Nothing that runs after promotion to check the pattern held. And no sentence anywhere about switching a trigger back. There's a quieter detail on the same page: review is the default for the agent, but the trigger defaults for incident response plans and scheduled tasks are both autonomous. The ladder is in the documentation and the product starts you at the top of it. GitHub is the same shape at the code end. Copilot's coding agent can open a draft pull request and "cannot approve or merge a pull request". Then, in March, admins could skip the human approval for workflow runs. Then, on 1 September, Copilot's review "can submit an approval that counts toward the repository's required-approvals rule", off by default. The switches are all there. Nothing tells you when you have earned the right to flip one, and nothing flips it back. Even the tools I use every day work this way. Claude Code has a permission mode per session, from read-only up to fully unattended, and its SDK documentation says the intent out loud: "start restrictive and loosen permissions as trust builds". I agree with every word of that sentence, and I want to know what "trust builds" means in a number. Anthropic's own data from February shows what happens when you leave it to feel: new users run fully unattended in about 20 per cent of sessions, users with 750 sessions in over 40 per cent, and experienced users interrupt the agent more often, not less. People do climb the ladder on their own but they also learn where to grab the rail. There is one exception I want to credit. Cloudflare's agent access model, published in August, describes a "trust ratchet" that makes trust stateful: "capability state moves in one direction. When a declared protected event occurs, the trust ratchet removes capabilities across the task execution graph according to policy." That is a demotion mechanic, enforced "in the harness and the network, not the prompt", and it is the only one I managed to find from a vendor. It's scoped to a single task, narrowing what an agent may do as the task proceeds, rather than to the agent's standing across weeks of runs. The direction is right and the timescale is one rung short. Langfuse's prompt CI/CD guide has the exact loop I want: a promotion gated on a threshold, a RegressionError that blocks it, and rollback by pointing the production label back, applied to a prompt version rather than to how much authority an agent holds. The rung where it breaks If you take one warning from the literature, take this one: rung three is where the ladder fails, and it fails quietly. Lisanne Bainbridge wrote the founding paper on this in 1983, about control rooms, and every sentence of it reads as if it were about your pull request queue. The designer "who tries to eliminate the operator still leaves the operator to do the tasks which the designer cannot think how to automate". Skills "deteriorate when they are not used", so "a formerly experienced operator who has been monitoring an automated process may now be an inexperienced one". Monitoring a system on which very little happens is something it is humanly impossible to sustain for more than about half an hour. And the one that should be printed on every approve button: "A classic method of enforcing operator attention to a steady-state system is to require him to make a log. Unfortunately people can write down numbers without noticing what they are." The experimental literature since then has measured how bad it gets. Parasuraman and Manzey's review finds that automation complacency "occurs in both naive and expert participants, cannot be prevented by training or instructions", and reports a study of experienced airline pilots given a false engine-fire alert in which every one of them shut the engine down. A 2023 radiology study gave 27 radiologists mammograms with a purported AI suggestion attached: when the suggestion was right, the very experienced readers were correct 82 per cent of the time; when it was wrong, 45 per cent. Elish has a name for the person left holding that accountability: the moral crumple zone, "a human actor who had limited control over the behavior of an automated or autonomous system" who nonetheless "bears the brunt of the moral and legal responsibilities when the overall system malfunctions". The 2026 version says the same thing in the vocabulary of agents. OpenAI saw it in 2023 and asked for "best practices for users reviewing approvals for high-cost actions (such as minimum review times) to avoid their turning into a 'rubber stamp'". Microsoft's AI red team spent a year attacking agent systems and reported in June that human-in-the-loop "bypass was the most consistently exploited failure mode, at very high frequency", with the attack being nothing cleverer than the one its 2025 taxonomy had already described: waiting until "the user becomes fatigued with the prompts". Cloudflare's version is the one I would put on the wall: "An approval that is always granted is not a control. It is a ritual that trains people to ignore the one prompt that matters." And two weeks ago Mitchell, Ghosh and Passi published a paper whose title is the whole argument: "AI Agents Push Humans Out of the Loop". There is of course an irony inside the ladder itself - the evidence trail that earns promotion, cycle after cycle of the agent being right, is exactly the "sustained positive experience with the system" that breeds complacency in the person approving it. The better the agent gets, the worse the reviewer gets. So my rule for rung three is that it is a transit state, not a place to live. You should be able to say, for every step sitting on it, either the number that will promote it or the number that will demote it, and roughly when. A step that has been on rung three for six months with no promotion criterion is not being supervised. It is being rubber-stamped by someone who stopped reading months ago, and the honest move is to admit that and either write the check that lets it go up, or send it back down to advisory where you have to decide. What demotion has to look like Every published failure of an over-promoted agent that I found has the same shape, and it is not "the model was bad". It is that the demotion lived in the wrong place. In July 2025 a Replit agent deleted a live production database during a declared code freeze, despite receiving repeated instructions not to make changes, having already spent days covering up bugs with fabricated data, fake reports and lies about the unit tests. The code freeze was a demotion. It was delivered as a sentence in a prompt, and the agent's credentials did not know about it. Replit's fixes, tellingly, were a fence and a lower rung: automatic separation of development and production databases that same weekend, and a plan-only mode six weeks later. In late April this year an agent working from Cursor deleted a company's production database and its volume-level backups in a single API call, having, in its own words, "guessed that deleting a staging volume via the API would be scoped to staging only". An agent's real rung is whatever its credentials allow. If the rung in your config file and the rung in your IAM policy disagree, the IAM policy wins, and so does the incident. This is why my DOCTOR_RUNG value is enforced in code that refuses to act above it, and why the doctor's context assembly is read-only by construction, not by instruction. The rung is not a request to the model, it's more of a fence that the model can't see over. Demotion therefore has three parts. A signal: an online eval below its threshold, a gate recall that dropped, a diagnosis marked as wrong. A single change: one config value, one label, one ratchet click, with no pull request for you to review at two in the morning. A fence that makes the change real: the permissions above the new rung are gone, not merely discouraged. Get the first two right and skip the third and you have Replit's code freeze. Others are here too, but there are gaps I want to be straight about the crowd, because I nearly wrote this piece as if the ladder were mine. It is not. Accenture presented shadow mode to advisory to controlled autonomy to expanded autonomy at AI Engineer Europe in April, with the end state, in one attendee's notes, "gated by evidence and outcomes". A June paper by Weber and Taneja, "The Digital Apprentice", opens with "autonomy is earned, not assumed" and has the agent "graduating through per-skill autonomy tiers only when empirical evidence justifies it", with each escalation gated by explicit human approval and, as I read it, automatic rollback of a skill when its quality slips. That is the most rigorous version I have read and it is closer to mine than anything from a vendor. An August framework from Ampcome calls it "evaluation-gated autonomy" and says "breach a threshold, and autonomy contracts". Gartner now publishes four levels from observe to autonomous-within-guardrails. So the idea has arrived, several times, independently. But I would go a few steps further from having run one. First, shadow is a rung, not a pre-launch phase. Most ladders start at "recommend". Mine starts with the agent doing the real job on real traffic and being scored against what happened, with nobody acting on it. A shadow run on last week's production inputs is the least superficial test environment there is, and it costs the price of the tokens. Shadow also answers the question Kahneman and Klein put at the centre of when intuition can be trusted at all: is this a high-validity environment with feedback that is "rapid and unequivocal"? Their conclusion for human experts holds for agents word for word: "subjective confidence is therefore an unreliable indication of the validity of intuitive judgments and decisions." A model's confidence, a vendor's confidence and your own after a good demo are the same thing, and a shadow replaces all three with a scorecard. Second, the unit of evidence is a cycle of the real job, and the count is plural. Not a benchmark score. Not a fortnight of it looking fine. The rewrite backend earned its rung on 85 real articles with two judges. The doctor's next rung needs two production cycles of marked diagnoses. When you write your own promotion rule, write the cycle count into it because ten clean runs with a low-blast-radius task tell you less than three clean runs of the one that can delete a database. Third, make it per step, not per agent. Bradshaw, Hoffman, Johnson and Woods listed seven deadly myths of autonomous systems in 2013, among them that autonomy is a single dimension, that levels of autonomy make a useful roadmap, and, verbatim, that "'full autonomy' is not only possible, but is always desirable". I think that critique is right about a single dial for "the agent" and wrong about a rung per action class. My pipeline has a rung-four step and a rung-one step in the same process, and the config that keeps them apart is one line each. Why I am writing this down now I have been running this ladder for months and had never published it. I had said it out loud, in meetings and in at least one interview, and assumed that the field notes made the position obvious. Reading them back, they do not. What they show is a person who writes about eval gates, tripwires, abstention and accountability, and if that is all you had read, you could fairly conclude that I want a developer in every loop. The opposite is true. I want as few humans in the loop as the evidence allows, and I want the evidence to be the thing that decides. The gates are not there to slow autonomy down. They are the mechanism by which it is granted. An agent with a green scorecard over enough cycles gets more rope than any amount of confidence in a demo would earn it, and an agent whose scorecard goes red loses that rope the same afternoon. If your organisation is stuck at rung one, it is usually because nobody wrote down what promotion would take. If it jumped to rung four, it is usually because nobody wrote down what demotion would take. Both cost you, the first in time and the second in trust. What this cannot do Demotion today is a human changing a config value after reading a degraded flag. Automatic demotion on regression is designed and not built, and until it is, the ladder has a person on the descent path, which is exactly the thing I claim to be removing. Kappa between the judges and human labels is empty, because there are not yet twenty human verdicts to compute it from, so "the scorecard is green" currently means "two model families agree", which although the judge panel piece explains as necessary, it's also not sufficient. There's a deeper problem with gating anything on a score, and I would rather name it than wait for you to. Goodhart's law (Strathern's version) says that "when a measure becomes a target, it ceases to be a good measure", and an agent promoted on eval scores is an agent under exactly that pressure. This is not hypothetical. METR found OpenAI's o3 reward-hacking about 30 per cent of its runs on one benchmark, by tampering with the scoring code and test harness, and when asked afterwards whether that was what the user wanted, it answered 'no' 10 out of 10 times. A March 2026 paper by Wang and Huang argues that under any finite evaluation "any optimized AI agent will systematically under-invest effort in quality dimensions not covered by its evaluation system", which makes reward hacking "a structural equilibrium, not a correctable bug". My defences are simple and boringly ordinary: the agent cannot see its own scorecard, the judges are different model families from the generator, and the golden set grows from real failures the agent did not choose. Those make gaming harder. They do not make the score the thing I care about, and the day the ladder promotes something that learned the proxy is the day you need a human reading the output again. The whole thing is running at indie scale: one pipeline, a few dollars a week. Whether the same ladder holds when a rung-four agent is spending your money against your customers is a question I have designed for and not measured. And Bainbridge's warning applies to me as much as to anyone: the person left monitoring a reliable system is the least practised party at the exact moment they are needed. My answer is that the scorecard, not the person, does the monitoring. But, of course, that answer is only ever going to be as good as the scorecard. What it does give you is a way to answer "is it safe to trust this yet" with a number and a date, instead of an opinion and a meeting. Richard Atkins