Skip to content

Where Should the Human Check Go?

Working notes

Current understanding of an active question, not a finished reference. Revised as the research develops.

When AI does several steps of a job in a row, where should a person check the work? The question came out of a brief on the paper Chaining Tasks, Redefining Work. In that paper's model, a person checks only the final output of a run of AI steps. Checking then costs the same however long the run is (Demirer et al., 2026). As I read it, that arrangement pays off only if the final check catches mistakes made earlier.

This site already covers two neighboring questions. When to Let AI Do It sets how much of a single task to hand to AI. The Four Levels of Workflow AI Integration explains why errors compound across steps. Neither says where in a sequence of steps the check belongs.

In brief, the evidence says more about where to put a check than about whether one check is enough. Established fields put the check just before the step that commits, where an error becomes costly or permanent. Studies of people reviewing AI output show they often accept its mistakes. That gives reason to doubt that one check at that point catches earlier errors. No study I found tests whether it does.

Errors build on each other

The simple picture treats each step as an independent chance of failure. Recent measurements suggest errors are not independent. Once a model's record of its own earlier steps contains mistakes, its later steps get less accurate (Sinha et al., 2026). In that study, recent models with built-in reasoning did not show this decline. Prompting other models to reason step by step did not stop it.

When a request arrives piece by piece over a conversation, models average 39% worse than when given it at once (Laban et al., 2025). After a wrong turn, they tend not to recover. Reliability also falls quickly as tasks get longer. On software tasks, agents reach 80% success only on tasks a quarter to a sixth as long as those they manage at 50% (METR, 2025).

Mistakes inherited from earlier steps often go unnoticed. One test gave agents that operate software screens a task that already contained a mistake. Across four proprietary AI agents, success fell by 39% to 52% the more steps had run since the mistake (Bu et al., 2026). The agents noticed the mistake 34% to 65% of the time (Bu et al., 2026). The authors work at Alibaba, which makes some of the models tested.

AI checkers can often tell that a run went wrong, but rarely find where. Reviewing finished runs with no special instructions, GPT-5.4 found the first error in 14.9% of runs and 5% of hard ones (Wang et al., 2026). One company tested an automated checkpoint inside its own agent on a public benchmark. It caught 8 of 40 wrong outputs, one in five, with no false alarms (Leni, 2026). The authors evaluated their own company's system. As I read them, neither result shows an AI checker reliably tracing where a run went wrong.

Checking every step can stop errors from building. One system completed over a million steps with zero errors (Meyerson et al., 2025). It split the work into single steps, had the AI answer each several times, and took a vote. Its task was a puzzle in which every step has one known right answer. Most business steps do not.

In practice, many deployed agents hand back to a person quickly. In a survey of 86 practitioners, 68% of deployed agents took fewer than 10 steps before a person stepped in (Pan et al., 2026).

Reasons to doubt a single final review

One check at the end assumes the reviewer catches what went wrong upstream. Studies of how people review and rely on AI output give reasons for doubt. None of them tests a review at the end of a chain of AI steps; they test review of single outputs.

  • Reviewers absorb the AI's errors. In a randomized trial, physicians trained in using AI could consult an AI chatbot on six diagnostic cases. Those whose advice had errors planted in three of the six scored 73.3% on diagnostic reasoning, against 84.9% with error-free advice (Qazi et al., 2025). Their top-choice diagnoses were right 76.1% of the time, against 90.5% (Qazi et al., 2025). The trial has since appeared in the peer-reviewed journal NEJM AI.
  • Their edits drift toward the AI's draft. Oncologists' GPT-4-assisted replies to patients ended up closer in content to the AI drafts than to replies they wrote alone (Chen et al., 2024). The authors suggest the doctors may have taken on the model's assessment.
  • Reviewers can feel sure and be wrong. People using an AI coding assistant wrote less secure code and believed it was more secure (Perry et al., 2023).
  • Longer explanations persuade. Longer AI explanations raised people's confidence without raising accuracy (Steyvers et al., 2025).
  • Review often goes unrecorded. On GitHub, a site where programmers share code, 61.4% of 33,596 code changes submitted by AI agents had no recorded review (Duma et al., 2026).
  • Skill can fade. Doctors used to an AI aid detected precancerous growths less often in colonoscopies done without it. Their detection rate fell from 28.4% to 22.4% (Budzyń et al., 2025).

How the work is shown and overseen matters. People checked step-by-step AI solutions to simple math problems, some right and some with planted errors (Zhou, Nguyen et al., 2026). They judged them correctly, including finding any faulty step, 73.5% of the time as plain text and 85.6% with the best interactive display. For AI agents working in a web browser, plan-based oversight cut the odds that a planted problem happened by 76% (Chen et al., 2026). Once a problem reached execution, people blocked only 29 of 138, about 21%, with no clear difference between setups (Chen et al., 2026).

Only one study I found compared checking along the way with checking at the end. Of 48 participants, 81% preferred checking along the way (Zhou, Roy et al., 2026). It did not measure errors caught.

Where established fields and AI guidance put the check

Fields that have run high-consequence processes for decades agree on a few principles, and recent AI guidance follows them. None of the sources below tests whether its placement of checks improves results.

  • By consequence, not at every step. Human factors is the study of how people work with machines. A founding framework in the field limits high automation of decisions and actions to low-risk situations (Parasuraman et al., 2000). Singapore's regulator, in non-binding guidance, asks for human approval before high-stakes or irreversible actions, and when an agent behaves unusually (IMDA, 2026). OpenAI's agent guide calls in a person for high-risk actions and when an agent exceeds its failure limits (OpenAI, 2025).
  • Just before the action that commits. The same framework adds a condition. Where the decision itself is highly automated, a person at the action stage is a "last chance opportunity" to trap errors (Parasuraman et al., 2000). Surgical teams pause before the first incision (WHO, 2009). In nuclear plant quality rules, "hold points" stop work until an inspector signs off (US NRC). In Anthropic's framework, its coding agent can read by default but must ask before changing code or systems (Anthropic, 2025).
  • A separate checker, and limits by amount. Banking controls require approval above set limits and keep approving a payment apart from making it. They also warn that controls treated as an add-on may be skipped under time pressure (Basel Committee, 1998).
  • Checks that fire on triggers. A UC Berkeley risk guide for AI agents suggests checks can fire after a set amount of activity, such as calls to other software. They can also fire when an agent is asked to act outside its remit (UC Berkeley CLTC, 2026).

Regulations leave placement open. For high-risk systems, the EU AI Act requires oversight in proportion to risk and autonomy, and a way to override or stop the system. It requires confirmation by two people only for systems that identify people remotely by their physical features (EU AI Act, 2024). The US standards body NIST asks organizations to define their oversight processes and says some systems may not need it (NIST, 2023).

Two cautions apply. Deciding in advance which actions are consequential may itself be hard (Feng et al., 2025). Adding a checkpoint also adds engineering work (Google Cloud, 2026).

A note on sources: Anthropic, OpenAI and Google all sell agents. I drafted this note with Claude, which Anthropic makes.

What checking costs

Every check takes time, and the evidence on how much is thin. One economic study argues that near-perfect AI accuracy costs disproportionately more, so keeping people on the remaining work is often cheaper (Li et al., 2026). It was calibrated on computer vision tasks. Another treats people's capacity to check AI work as the binding limit on growth (Catalini et al., 2026). In that model, deploying unchecked systems can pay for a single firm while hidden risk builds up. A third predicts that workers who are weaker at checking hand too much to AI and check less (Huang et al., 2026).

Measured costs are few. In a randomized trial, experienced developers took 19% longer with AI and spent about 9% of their time reviewing and cleaning its output (METR, 2025). In Denmark, chatbot users reported saving 2.8% of work hours on average (Humlum & Vestergaard, 2025). The same study found that chatbots created new tasks for 8.4% of workers, including reviewing AI output. Two vendor surveys of workers' estimates report that checking and fixing AI output uses up much of the time AI saves (Workday, 2026; Foxit, 2026).

Getting it wrong has a price too. A British Columbia tribunal held Air Canada liable for wrong information from its customer-facing chatbot (BC Civil Resolution Tribunal, 2024). The tribunal found the airline had not taken reasonable care to make sure the chatbot was accurate.

Where the evidence leaves the question

My reading, which the evidence does not settle:

  1. Check before any step that is costly or hard to undo. That means sending, paying, deleting, publishing or changing an important record. The established fields agree on this, and the AI guidance points the same way.
  2. Do not count on that check to catch a mistake made earlier. In the physician trial, errors planted in the AI's advice lowered doctors' diagnostic scores (Qazi et al., 2025). I expect the same at the end of a chain, though no study tests it. Where an early step feeds everything after it, check that step's output too.
  3. Make the check a required step the work cannot skip. Checks treated as extras can be skipped under time pressure. Where the stakes justify it, give the check to someone other than the person who ran the work.
  4. Check selectively, guided by consequence. Every check costs time, and the Workday and Foxit surveys report that checking uses up much of what AI saves (Workday, 2026; Foxit, 2026).

This is more cautious than the model behind Chaining Tasks, which assumes the final check works. Where checks go is one part of the wider question in Rethink the Process, or Just Add AI?.

Still open

  • Does an end-of-job review catch an early error hidden in a polished result? I found no study that tests it.
  • Does checking step by step catch more errors than checking at the end? The one comparison I found measured preference, not errors caught.
  • Where do organizations actually put their checks, and what happens? I found no firm-level evidence.
  • What does a check cost against the errors it prevents? No study I found measures both.
  • Does checking every step, as in the million-step puzzle study, work when a step has no single right answer?

Sources

  • Demirer, Horton, Immorlica, Lucier & Shahidi — Chaining Tasks, Redefining Work: A Theory of AI Automation (arXiv 2606.15960; NBER Working Paper 34859), 2026. Output verification is therefore a fixed cost of the chain rather than a marginal cost that scales with each additional step. View source · verified 2026-09-15 · primary
  • Sinha, Arun, Goel, Staab & Geiping — The Illusion of Diminishing Returns: Measuring Long Horizon Execution in LLMs (ICLR 2026; arXiv 2509.09677), 2026. as we increase the rate of injected errors into the context, accuracy at turn 100 consistently degrades further ... as models make mistakes, they become more likely to make more mistakes ... Qwen3 thinking models do not self-condition ... We find that recent thinking models are not affected by prior mistakes, fixing the self-conditioning effect ... with CoT prompting, finding that accuracy still deteriorates as the induced error rate increases. View source · verified 2026-09-15 · primary
  • Laban, Hayashi, Zhou & Neville (Microsoft Research, Salesforce Research) — LLMs Get Lost In Multi-Turn Conversation (arXiv 2505.06120), 2025. with an average drop of 39% across six generation tasks ... when LLMs take a wrong turn in a conversation, they get lost and do not recover. View source · verified 2026-09-15 · primary
  • METR (Kwa, West, Becker et al.) — Measuring AI Ability to Complete Long Software Tasks (arXiv 2503.14499, v4 July 2026), 2025. models' 80% time horizons are 4-6x shorter. View source · verified 2026-09-15 · primary
  • Bu, Liu, Chen, Jiang, Li et al. (Alibaba Cloud Computing) — Recovering Policy-Induced Errors: Benchmarking and Trajectory Synthesis for Robust GUI Agents (arXiv 2605.29447), 2026. performance drops as error depth increases, likely because the environment drifts further from the goal ... whether the agent recognizes the error immediately after takeover ... All authors are employees of Alibaba Cloud Computing. View source · verified 2026-09-15 · primary
  • Wang, Feng, Wu et al. (JIUTIAN Research, Nanjing University) — Where Do Deep-Research Agents Go Wrong? Span-Level Error Localization in Agent Trajectories (TELBench; arXiv 2606.02060, v2), 2026. still struggle to pinpoint the earliest annotated error span ... failed trajectories place substantially more first errors in the earliest and latest position bins ... Current auditors can often detect that a trajectory has become unreliable. View source · verified 2026-09-15 · primary
  • Dastidar and the Leni Team (Leni Inc.) — Where Does Agent Reliability Come From? A Cross-Benchmark Decomposition of Verification Loops, Specialist Models, and Scaffolding in a Production Enterprise Agent (arXiv 2607.17044), 2026. catch rate c = 8 / 40 = 0.20 ... This is a vendor evaluation of the vendor's own system. View source · verified 2026-09-15 · primary
  • Meyerson, Paolo, Dailey, Shahrzad, Francon et al. (Cognizant AI Lab, UT Austin) — Solving a Million-Step LLM Task with Zero Errors (arXiv 2511.09030), 2025. solves a task with over one million LLM steps with zero errors ... the impact on correlated errors turns out to be a much more important effect ... error correction based on subtask-level voting. View source · verified 2026-09-15 · primary
  • Pan, Arabzadeh, Cogo et al. — Measuring Agents in Production (arXiv 2512.04123, v4 June 2026), 2026. 68% of deployed agents execute fewer than 10 steps before requiring human intervention ... surveyed 86 deployed systems practitioners. View source · verified 2026-09-15 · primary
  • Qazi, Ali, Khawaja, Akhtar, Sheikh & Alizai — Automation Bias in Large Language Model Assisted Diagnostic Reasoning Among AI-Trained Physicians (medRxiv 2025.08.23.25334280; published in NEJM AI 3(5), 2026), 2025. achieved mean (SD) diagnostic accuracy of 84.9% (19.7%), whereas those exposed to flawed recommendations scored 73.3% (30.5%) ... resulting in an adjusted mean difference of -14.0 percentage points ... the treatment group's recommendations contained deliberate errors in 3 of 6 vignettes ... Top-choice diagnosis accuracy per case was 76.1% (42.5) in the treatment group and 90.5% (28.9) in the control group. View source · verified 2026-09-15 · primary
  • Chen, Guevara, Moningi et al. — The effect of using a large language model to respond to patient messages (The Lancet Digital Health 6(6)), 2024. posed a risk of severe harm in 11 (7.1%) of 156 survey responses ... The content of the LLM-assisted responses were more similar to the LLM drafts ... physicians might not simply use LLMs to better phrase their own assessment, but instead adopt the assessment by the LLM. View source · verified 2026-09-15 · primary
  • Perry, Srivastava, Kumar & Boneh (Stanford) — Do Users Write More Insecure Code with AI Assistants? (ACM CCS 2023), 2023. believed their answers were more secure than those in the control group despite often providing more insecure answers. View source · verified 2026-09-15 · primary
  • Steyvers, Tejeda, Kumar et al. (UC Irvine) — What large language models know and what people think they know (Nature Machine Intelligence 7), 2025. longer explanations increased user confidence, even when the extra length did not improve answer accuracy. View source · verified 2026-09-15 · primary
  • Duma, Wróblewski, Bobińska, Winiarska & Przymus — These Aren't the Reviews You're Looking For: How Humans Review AI-Generated Pull Requests (EASE 2026), 2026. 61.38% (20,621) receive no recorded review activity. View source · verified 2026-09-15 · primary
  • Budzyń, Romańczyk, Kitala et al. — Endoscopist deskilling risk after exposure to artificial intelligence in colonoscopy: a multicentre, observational study (The Lancet Gastroenterology & Hepatology), 2025. decreased significantly from 28.4% (226 of 795) before to 22.4% (145 of 648) after exposure to AI. View source · verified 2026-09-15 · primary
  • Zhou, Nguyen, Kharya, Nguyen & Agarwal — Improving Human Verification of LLM Reasoning through Interactive Explanation Interfaces (IUI 2026), 2026. the iGraph interface yielded the highest clarity and error detection rate (85.6%) ... all outperforming standard CoT (73.5%) ... both correct and incorrect. View source · verified 2026-09-15 · primary
  • Chen, Zhang, Chen et al. — Comparing Human Oversight Strategies for Computer-Use Agents (arXiv 2604.04918, v2), 2026. oversight strategy affected exposure to problematic actions, but not users' ability to intervene ... plan-based strategies reduced the odds that a problematic action occurred by 76% ... users blocked only 29 of the 138 problematic actions that reached execution (21%), with no reliable difference across strategies ... regardless of whether the user noticed it. View source · verified 2026-09-15 · primary
  • Zhou, Roy, Gupta et al. (Georgia Institute of Technology) — When Should Users Check? Modeling Confirmation Frequency in Multi-Step Agentic AI Tasks (CHI 2026), 2026. Results show that 81% of participants (39 of 48) preferred our intermediate confirmation strategy over the confirm-at-end baseline. View source · verified 2026-09-15 · primary
  • Parasuraman, Sheridan & Wickens — A Model for Types and Levels of Human Interaction with Automation (IEEE Transactions on Systems, Man, and Cybernetics, Part A, 30(3)), 2000. high levels should be implemented only for low-risk situations ... last chance opportunity ... if the previous decision selection stage has been highly automated. View source · verified 2026-09-15 · ⚠ secondary mirror
  • Infocomm Media Development Authority (IMDA), Singapore — Model AI Governance Framework for Agentic AI, Version 1.5, 2026. Organisations should define significant checkpoints or action boundaries that require human approval ... Irreversible actions e.g. permanently deleting data, sending communications, making payments ... Outlier or atypical behaviour. View source · verified 2026-09-15 · primary
  • OpenAI — A Practical Guide to Building Agents, 2025. Two primary triggers typically warrant human intervention ... Actions that are sensitive, irreversible, or have high stakes should trigger human oversight. View source · verified 2026-09-15 · primary
  • World Health Organization — Implementation Manual WHO Surgical Safety Checklist 2009 (Safe Surgery Saves Lives), 2009. Before making the first surgical incision, a momentary pause should be taken by the team in order to confirm that several essential safety checks are undertaken. View source · verified 2026-09-15 · ⚠ secondary mirror
  • US Nuclear Regulatory Commission — 10 CFR Part 50, Appendix B, Criterion X (Inspection), current. beyond which work shall not proceed without the consent of its designated representative are required, the specific hold points shall be indicated in appropriate documents. View source · verified 2026-09-15 · ⚠ secondary mirror
  • Anthropic — Our framework for developing safe and trustworthy agents, 2025. must ask for human approval before taking any actions that modify code or systems. View source · verified 2026-09-15 · primary
  • Basle Committee on Banking Supervision — Framework for Internal Control Systems in Banking Organisations, 1998. Requiring approval and authorisation for transactions over certain limits ensures that an appropriate level of management is aware of the transaction ... When controls are viewed as an addition to the day-to-day activities, they are often seen as less important and may not be performed in situations where individuals feel pressured to complete activities in a limited amount of time ... An effective internal control system requires that there is appropriate segregation of duties. View source · verified 2026-09-15 · primary
  • UC Berkeley Center for Long-Term Cybersecurity — Agentic AI Risk-Management Standards Profile, Version 1.0, 2026. Quantitative trigger points (e.g., duration of unsupervised activity, number of API calls) ... Qualitative trigger points (e.g., requests outside of the agent's predefined scope. View source · verified 2026-09-15 · primary
  • European Union — Regulation (EU) 2024/1689 (AI Act), Article 14: Human oversight, 2024. The oversight measures shall be commensurate with the risks, level of autonomy and context of use of the high-risk AI system ... or to otherwise disregard, override or reverse the output of the high-risk AI system ... interrupt the system through a 'stop' button or a similar procedure that allows the system to come to a halt in a safe state ... separately verified and confirmed by at least two natural persons. View source · verified 2026-09-15 · primary
  • NIST — Artificial Intelligence Risk Management Framework (AI RMF 1.0), NIST AI 100-1, 2023. Processes for human oversight are defined, assessed, and documented ... Some AI systems may not require human oversight. View source · verified 2026-09-15 · primary
  • Feng, McDonald & Zhang (University of Washington; published with the Knight First Amendment Institute) — Levels of Autonomy for AI Agents (arXiv 2506.12469), 2025. reliably determining which actions are consequential may be challenging. View source · verified 2026-09-15 · primary
  • Google Cloud — Choose a design pattern for your agentic AI system (Cloud Architecture Center), 2026. This pattern can add significant architectural complexity. View source · verified 2026-09-15 · primary
  • Li, Aboutorabi, Lyu, Qian, Fleming, Goehring & Thompson (MIT, EPFL, IBM Research, IBM Institute for Business Value) — Economics of Human and AI Collaboration: When is Partial Automation More Attractive than Full Automation? (arXiv 2603.29121), 2026. Full automation is therefore often not cost-minimizing ... can be orders of magnitude more costly. View source · verified 2026-09-15 · primary
  • Catalini, Hui & Wu — Some Simple Economics of AGI (arXiv 2602.20946), 2026. It is human verification bandwidth ... deploying unverified systems becomes privately rational ... the binding constraint on growth is no longer intelligence ... hidden debt accumulates in the gap between visible metrics and actual human intent. View source · verified 2026-09-15 · primary
  • Huang, Xiao & Vishnoi — Delegation and Verification Under AI (ICML 2026; arXiv 2603.02961), 2026. rationally over-delegate and reduce oversight. View source · verified 2026-09-15 · primary
  • METR (Becker, Rush, Barnes & Rein) — Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (arXiv 2507.09089), 2025. developers spend approximately 9% of their time reviewing and cleaning AI generated outputs ... allowing AI actually increases completion time by 19%. View source · verified 2026-09-15 · primary
  • Humlum & Vestergaard — Large Language Models, Small Labor Market Effects (BFI Working Paper 2025-56; NBER w33777), 2025. AI chatbots have created new job tasks for 8.4% of workers ... Reviewing AI-generated content for accuracy, clarity, and relevance. The human fact-checks, corrects errors ... users report average time savings of just 2.8% of work hours. View source · verified 2026-09-15 · primary
  • Workday — New Workday Research: Companies Are Leaving AI Gains on the Table (Beyond Productivity: Measuring the Real Value of AI), 2026. Nearly 40% of AI time savings are lost to rework. View source · verified 2026-09-15 · primary
  • Foxit — The State of Document Intelligence (press release), 2026. Executives believe AI saves them an average of 4.6 hours per week, yet they spend 4 hours and 20 minutes per week validating AI-generated outputs. View source · verified 2026-09-15 · primary
  • British Columbia Civil Resolution Tribunal — Moffatt v. Air Canada, 2024 BCCRT 149, 2024. did not take reasonable care to ensure its chatbot was accurate ... responsible for all the information on its website. View source · verified 2026-09-15 · primary