Nine Fallacies of Professional AI Usage: An Evidence-Based Synthesis

1. Introduction

Generative AI tools have entered professional work in law, consulting, software engineering, customer service and many other fields. Much of the evidence is encouraging. in a field study of more than 5,000 customer-support agents, AI assistance increased the number of issues resolved per hour by 15% on average [1]. in a preregistered experiment, professionals completed writing tasks in 40% less time and at higher quality [2]. Consultants using GPT-4 on suitable tasks worked faster and produced better results [3].

Other findings point the other way. The same consulting experiment found that consultants using AI were less likely to reach correct answers on a task outside the tool’s capability [3]. Experienced developers working on their own codebases took longer with AI tools while believing they were faster [4]. Knowledge workers who trusted AI more reported less critical thinking [5], and developers learning a new library with AI assistance acquired less of the underlying skill [6]. Courts in the United States and in England and Wales have sanctioned or criticized lawyers for fabricated AI output [7,8], and a Canadian tribunal held a company liable for its chatbot’s statements [9].

Read together, these findings show that the effect of AI on professional work depends on conditions that users often do not see. in each negative case, an assumption that seemed reasonable when the tool was introduced turned out to be false under conditions of professional use. Distributed computing faced a comparable problem in the 1990s. Developers building networked systems made assumptions that held in the test environment and failed in production. L. Peter Deutsch and colleagues captured these assumptions in a short list, the Fallacies of Distributed Computing, which remains in use three decades later. This paper borrows the format of that list, a small set of plainly stated assumptions, and applies it to a different problem with different content.

No comparable list for professional AI usage was found (Section 2.5). Existing fallacy lists for AI address expectations within AI research [10] or the assumption that deployed systems function at all [11]. The human factors literature offers an established body of theory on automation [12,13,14] and has recently been applied to generative AI [15]. These strands have not been combined into a framework that practitioners and organizations can check their own use against.

This paper addresses the following research question: which recurring false assumptions about AI usage in professional settings are supported by current evidence, under which conditions do they fail, and how do they interact?

The paper makes three contributions:

1.

It proposes nine fallacies of professional AI usage. For each, it explains why the assumption persists, presents the evidence against it, states its boundary conditions and grades the strength of the evidence (Table 2).

2.

It analyzes how the fallacies can compound. It identifies a specific risk for software systems built largely by coding agents, which may become unmaintainable when access to the agent is lost.

3.

It derives implications for practice, a research agenda and a self-assessment instrument.

Section 2 reviews related work. Section 3 describes the method. Section 4 presents the nine fallacies. Section 5 discusses their interactions, implications for practice, a research agenda and limitations, and Section 6 concludes.

2. Background and Related Work

2.1. Source of Inspiration: The Fallacies of Distributed Computing

In 1994, L. Peter Deutsch, then a Fellow at Sun Microsystems, drafted seven assumptions that designers of distributed systems are likely to make and that prove wrong over time, and James Gosling added an eighth in 1997 [16]. The eight fallacies are: the network is reliable; latency is zero; bandwidth is infinite; the network is secure; topology does not change; there is one administrator; transport cost is zero; and the network is homogeneous.

The list has three properties that make it a useful model. Each item is phrased as a plain statement that practitioners would not consciously endorse but nonetheless act upon. Each can be checked against a concrete design. and each failure tends to surface only after deployment, when it is expensive to correct. Rotem-Gal-Oz [16] later explained each fallacy with its causes, consequences and countermeasures. The present paper takes this format as its source of inspiration: a short list of plainly stated assumptions, each explained in a fixed structure. Its content, its five-part structure (Section 3.6) and its evidence grading are independent of the distributed computing list.

2.2. Fallacy Lists and Traps in AI Research

Mitchell [10] identifies four fallacies in common assumptions of AI researchers that lead to overconfident predictions about the field. One of them, which Mitchell calls wishful mnemonics, concerns benchmark names that suggest broader abilities than the benchmarks measure. Mitchell’s list addresses expectations within AI research and does not cover workplace use.

Raji et al. [11] argue that many deployed AI systems do not work as claimed, and that scholars, the press and policymakers pay too little attention to functionality. They call this the fallacy of AI functionality and present a taxonomy of known functionality failures. Their focus is AI policy, and their case studies largely predate the spread of generative AI.

Selbst [17] describe five traps that fairness-aware machine learning can fall into: the framing, portability, formalism, ripple effect and solutionism traps. Their analysis concerns system design for fairness. The portability trap is directly relevant to professional use (Section 4.8).

2.3. Human Factors of Automation

Bainbridge [12] argued that automating industrial processes can expand the problems faced by human operators. Operators are left with the tasks that could not be automated, must monitor systems for long periods, and lose practice in the skills they need when automation fails. Lee and See [13] frame appropriate reliance on automation as a matter of trust calibration: the correspondence between a person’s trust in an automated system and the system’s actual capabilities. Overtrust, where trust exceeds capability, leads to misuse. Parasuraman and Manzey [14] integrated research on automation complacency and automation bias. They found that both affect novices and experts, and that automation bias is not prevented by training or instructions.

Simkute et al. [15] apply this tradition to generative AI. Drawing on human factors research and recent user studies, they identify four sources of productivity loss: a shift from production to evaluation, unhelpful restructuring of workflows, interruptions, and a tendency for automation to make easy tasks easier and hard tasks harder.

2.4. Empirical Studies of AI-assisted Professional Work

A growing body of empirical studies examines AI in professional settings, with mixed results. Several studies report substantial gains. Brynjolfsson et al. [1] found a 15% average increase in issues resolved per hour among 5,172 customer-support agents, with the largest gains for less experienced and lower-skilled workers. Noy and Zhang [2] found that 453 college-educated professionals completed occupation-specific writing tasks in 40% less time and with 18% higher quality. Peng et al. [18] found that developers with GitHub Copilot implemented an HTTP server 55.8% faster than a control group. Pooling three field experiments with 4,867 developers, Cui et al. [19] report a 26.08% increase in completed tasks, with larger gains for less experienced developers.

Other studies report limits and losses. Dell’Acqua et al. [3] describe a jagged technological frontier of AI capability in an experiment with 758 consultants. Becker et al. [4], working at METR, report a randomized trial in which experienced developers took longer with AI tools. Lee et al. [5] surveyed 319 knowledge workers on critical thinking with generative AI, and Shen and Tamkin [6] measured skill formation in a randomized experiment with 52 developers. These studies are discussed in Section 4. Read together, the two groups of studies suggest that the effect of AI depends on the task, the worker’s experience and the way the tool is used, which is the premise of this paper.

2.5. Gap

These literatures address parts of the problem in different vocabularies. No framework was found that consolidates the evidence into a short list of false assumptions about professional AI usage, with the conditions under which each fails. One practitioner blog post [20] proposes eight fallacies of generative AI development, modeled on Deutsch’s list. It is limited to software engineering and is not peer reviewed, so it is treated here as prior art only.

3. Method

3.1. Approach

This is a conceptual paper based on an integrative review. Torraco [21] describes the integrative literature review as a distinctive form of research that generates new knowledge about the topic reviewed. Snyder [22] characterizes its purpose as assessing, critiquing and synthesizing the literature on a topic so that new theoretical frameworks and perspectives can emerge. This approach suits a topic whose evidence is spread across disciplines and includes legal and operational records alongside academic studies. The review was conducted in five steps: scoping, search, screening and verification, derivation of the fallacies, and evidence grading.

3.2. Scope

Professional AI usage is defined as the use of generative AI and AI decision support by individuals and organizations to carry out work, such as drafting, analysis, research, coding, customer service and decision preparation. The development and training of models are out of scope, except where they affect users. Sources in English were considered, together with primary records in other languages where needed [23]. No restriction on publication date was applied.

3.3. Search Strategy

Searches were conducted in October 2026 using a general web search engine and direct queries of arXiv, the ACM Digital Library, SSRN, publisher websites, Crossref metadata records, court databases and published copies of judgments, EUR-Lex, regulator websites and AI provider documentation. They were organized in five strands:

1.

existing lists of fallacies, traps or ironies concerning AI and automation;

2.

human factors research on automation bias, trust and skill retention;

3.

controlled studies of AI-assisted professional work, including studies reporting productivity gains, which were sought deliberately to establish boundary conditions;

4.

court decisions, regulator decisions and legislation on liability for AI output and on access to AI services;

5.

provider incident reports and documentation on service availability and model retirement.

Search terms combined the concepts of fallacies, assumptions, automation bias, overreliance, trust, productivity, skill formation, accountability and availability with generative AI and professional work. Backward reference searching was applied to key sources; for example, the gain studies cited in Section 4 were identified through the reference list of Shen and Tamkin [6] and then retrieved from their original publications. Searching for a given fallacy ended once at least one admissible source showed the assumption to fail and studies reporting contrary results had been checked for its boundary conditions. Subscription databases such as Scopus and Web of Science were not searched, so the review is not systematic.

3.4. Inclusion Criteria

Evidence was admitted in three categories:

1.

peer-reviewed journal articles and conference papers;

2.

randomized or preregistered studies, including preprints and working papers, which are identified as such;

3.

primary records, namely court decisions, regulator decisions, legislation, provider incident reports and provider documentation.

Grey literature, such as blog posts, opinion pieces and press reports, is cited only to document prior art or history or, where labeled, as an illustration. It is not used as evidence for any fallacy.

3.5. Screening and Verification

Each candidate source was opened at its primary location where possible: the publisher page or DOI record, the arXiv or SSRN record, the official court or legislative database, or the provider’s own page. Bibliographic details were checked against the publisher record, and every figure quoted in this paper was checked against the source text. Two decisions illustrate the screening. A figure of 34% for novice workers, reported in the working-paper version of Brynjolfsson et al. [1], was not used because it could not be confirmed against the published version. A press report on manual workstations at Toyota was retained only as a labeled illustration because it contains no measurement. Where a primary source could not be accessed, the reference-list entry says so [24]. Table 1 shows the composition of the resulting evidence base.

3.6. Derivation of the Fallacies

Candidate fallacies were formulated iteratively. An initial set was derived from the literature in the first three search strands. Two further candidates originated in the author’s professional experience and were then tested against the evidence like the others: the loss or reduction of access to AI models (Fallacy 9), and the missing mental model of software systems built largely by coding agents (Fallacy 7, Section 4.7.2). A candidate was retained if it met three conditions. First, it can be stated as a plain assumption that a practitioner might act on. Second, at least one admitted source shows the assumption to be false in a professional setting. Third, it is distinct from the other fallacies in its mechanism, even where fallacies interact. The missing mental model of agent-built systems did not meet the second condition on its own, because no study measures it directly. It was therefore retained as a component of Fallacy 7, where its status as an inference is stated (Section 4.7.2).

Each fallacy in Section 4 is presented in five parts: the assumption; why it persists, drawing on established theory where available; the evidence against it; its boundary conditions, meaning the circumstances in which the assumption does hold; and its consequences.

3.7. Evidence Grading

Each fallacy is assigned one of three evidence grades. Where a fallacy has distinct components or bodies of evidence, each is graded separately.

• Strong: consistent findings from multiple controlled studies, or a review that integrates such studies.
• Moderate: at least one controlled study of adequate size, or several independent primary records, with consistent findings on the core claim.
• Limited: a single small study, survey evidence, isolated cases, or inference from related evidence.

The grade refers only to the evidence that the assumption fails in professional settings; how often it fails is not graded. Grades are reported in Table 2. They are the author’s assessment and are open to revision as evidence accumulates.

3.8. Use of AI Tools

A generative AI model, Claude (model version claude-fable-5-1, Anthropic), with web search, was used during the search, screening, synthesis and drafting described above. Its role and the author’s role are set out in the declaration at the end of the paper. All included sources were verified against their primary locations as described in Section 3.5.

4. The Nine Fallacies

Table 2 summarizes the nine fallacies, their principal evidence and its grade. The subsections that follow treat each in turn. Fallacy 7 receives more space than the others because it contains the paper’s main original argument (Section 4.7.2).

Table 2. Fallacies of professional AI usage, principal evidence and evidence grade.

#	Fallacy	Counter-evidence in brief	Principal sources	Evidence grade
1	The output is correct	Fluent output can be fabricated; specialized tools also hallucinate	Magesh et al. [25]; Mata v. Avianca, Inc. [7]; Ayinde v London Borough of Haringey [8]	Moderate
2	Capability is uniform across tasks	Performance differs sharply between tasks of similar apparent difficulty	Dell’Acqua et al. [3]; Simkute et al. [15]	Moderate
3	The human reviewer is a reliable safeguard	Automation bias affects novices and experts and resists training	Parasuraman and Manzey [14]; Lee and See [13]; Lee et al. [5]	Strong (automation); Limited (generative AI)
4	Perceived productivity equals actual productivity	Measured time rose while developers believed they were faster	Becker et al. [4]	Limited
5	AI removes work	Work shifts toward evaluation, integration and handling interruptions	Simkute et al. [15]; Shen and Tamkin [6]; Bainbridge [12]	Moderate
6	Accountability moves to the tool or vendor	Courts and tribunals in three jurisdictions hold the deploying party responsible	Moffatt v. Air Canada [9]; Mata v. Avianca, Inc. [7]; Ayinde v London Borough of Haringey [8]	Moderate
7	Skills and system understanding stay intact when work is delegated	AI assistance can reduce skill formation; programs depend on human theory	Shen and Tamkin [6]; Casner et al. [35]; Naur [36]; Xia et al. [37]	Moderate (7a); Limited (7b)
8	Results transfer from benchmarks and pilots	Benchmark construct validity is often neglected; field and benchmark results diverge	Selbst [17]; Raji et al. [26]; Becker et al. [4]	Limited
9	The model is available	Access has been lost through outages, regulatory action, export controls and scheduled retirements	OpenAI [29]; Garante per la protezione dei dati personali [23]; Anthropic [31]; OpenAI [30]	Moderate

Table 2. Fallacies of professional AI usage, principal evidence and evidence grade.

#	Fallacy	Counter-evidence in brief	Principal sources	Evidence grade
1	The output is correct	Fluent output can be fabricated; specialized tools also hallucinate	Magesh et al. [25]; Mata v. Avianca, Inc. [7]; Ayinde v London Borough of Haringey [8]	Moderate
2	Capability is uniform across tasks	Performance differs sharply between tasks of similar apparent difficulty	Dell’Acqua et al. [3]; Simkute et al. [15]	Moderate
3	The human reviewer is a reliable safeguard	Automation bias affects novices and experts and resists training	Parasuraman and Manzey [14]; Lee and See [13]; Lee et al. [5]	Strong (automation); Limited (generative AI)
4	Perceived productivity equals actual productivity	Measured time rose while developers believed they were faster	Becker et al. [4]	Limited
5	AI removes work	Work shifts toward evaluation, integration and handling interruptions	Simkute et al. [15]; Shen and Tamkin [6]; Bainbridge [12]	Moderate
6	Accountability moves to the tool or vendor	Courts and tribunals in three jurisdictions hold the deploying party responsible	Moffatt v. Air Canada [9]; Mata v. Avianca, Inc. [7]; Ayinde v London Borough of Haringey [8]	Moderate
7	Skills and system understanding stay intact when work is delegated	AI assistance can reduce skill formation; programs depend on human theory	Shen and Tamkin [6]; Casner et al. [35]; Naur [36]; Xia et al. [37]	Moderate (7a); Limited (7b)
8	Results transfer from benchmarks and pilots	Benchmark construct validity is often neglected; field and benchmark results diverge	Selbst [17]; Raji et al. [26]; Becker et al. [4]	Limited
9	The model is available	Access has been lost through outages, regulatory action, export controls and scheduled retirements	OpenAI [29]; Garante per la protezione dei dati personali [23]; Anthropic [31]; OpenAI [30]	Moderate

4.1. Fallacy 1: The Output is Correct

Assumption Fluent, well-formatted output from an AI system is accurate.

Why it persists Generated output is grammatical, confident and formatted like expert work, whether or not it is correct. Research on processing fluency shows that people tend to associate fluency with truth: statements that are easy to process are judged as more likely to be true [38]. Language models produce fluent text by design, so the cue that readers use to judge credibility is present even when the content is wrong. Errors usually carry no visible signal.

Evidence Raji et al. [11] document that many deployed AI systems do not function as claimed, and that discourse on AI risk often assumes functionality without examining it. For specialized professional tools, Magesh et al. [25] conducted a preregistered evaluation of AI legal research tools whose providers had made strong reliability claims, using 202 queries. The tools from LexisNexis and Thomson Reuters each hallucinated between 17% and 33% of the time, which the authors conclude shows that the providers’ claims were overstated. In Mata v. Avianca, Inc. [7], attorneys filed a brief citing judicial opinions that did not exist, with fabricated quotations and citations generated by ChatGPT, and were sanctioned. In Ayinde v London Borough of Haringey [8], the High Court of England and Wales reviewed two cases in which lawyers had cited non-existent authorities, including one in which 18 of 45 citations referred to cases that did not exist. The court stated that freely available generative AI tools are not capable of conducting reliable legal research.

Boundary conditions The fallacy is compatible with AI output being correct in most cases. Inside the frontier of a tool’s capability, output quality can exceed unassisted work; consultants using GPT-4 on such tasks produced results of more than 40% higher quality than a control group [3]. The fallacy concerns the absence of a reliable signal that separates correct from incorrect output, and error rates differ between tools and tasks [25]. Where output can be checked cheaply and mechanically, for example code against an existing test suite, the cost of the fallacy is lower.

Consequences Fabricated or inaccurate content reaches clients, courts, codebases and decisions. Because the error is not visible in the output, it is detected late, if at all. The resulting liability is discussed under Fallacy 6.

4.2. Fallacy 2: Capability is Uniform Across Tasks

Assumption A tool that performs well on one task will perform well on tasks that appear similar.

Why it persists People judge task difficulty by human standards, and AI difficulty does not follow them. in the terms of Lee and See [13], appropriate reliance requires not only calibration but also resolution, meaning that trust distinguishes between levels of capability, and specificity, meaning that trust is tied to particular functions of a system. Where trust lacks specificity, trust formed on one kind of task extends to the tool as a whole, and early successes build trust that reaches beyond the tasks that justified it.

Evidence Dell’Acqua et al. [3] conducted a preregistered experiment with 758 consultants at Boston Consulting Group, about 7% of its individual-contributor consultants. On 18 realistic tasks inside the frontier of AI capability, consultants using GPT-4 completed 12.2% more tasks, worked 25.1% faster and produced results of more than 40% higher quality than a control group. On a task selected to lie outside the frontier, consultants using AI were 19 percentage points less likely to produce a correct solution. The authors describe the frontier as jagged: tasks of similar apparent difficulty fall on different sides of it. Simkute et al. [15] similarly note that automation tends to make easy tasks easier and hard tasks harder.

Boundary conditions Within a set of tasks known to lie inside the frontier, gains can be broad and consistent. in the same experiment, consultants below the average performance threshold improved by 43% and those above it by 17%, compared with their own baseline scores [3]. The fallacy applies when the boundary of that task set is unknown or when the tool is applied beyond it.

Consequences Trust formed on tasks inside the frontier carries over to tasks outside it, where AI use lowers the quality of results. Since the boundary is not visible to the user, the shift can go unnoticed.

4.3. Fallacy 3: The Human Reviewer is A Reliable Safeguard

Assumption Placing a human in the loop to review AI output provides sufficient control.

Why it persists Human review is an established quality control, and reviewers are often experienced professionals. Two mechanisms undermine it. First, automation bias and automation complacency: people tend to accept automated recommendations and to monitor reliable systems less closely, tendencies that Parasuraman and Manzey [14] attribute largely to the allocation of attention. Second, overtrust: when trust exceeds the system’s capability, people rely on it where they should not [13]. Bainbridge [12] added that monitoring is itself a demanding task, and that the more advanced an automated system is, the more crucial the operator’s contribution becomes.

Evidence Parasuraman and Manzey [14] reviewed empirical research on automation complacency and automation bias. Imperfect decision aids produced both omission and commission errors. Automation bias occurred in novices and experts, was not prevented by training or instructions, and affected teams as well as individuals. For generative AI, Lee et al. [5] surveyed 319 knowledge workers who reported 936 examples of AI use at work. Higher confidence in generative AI was associated with less critical thinking, and higher self-confidence with more. Critical thinking shifted toward verifying information, integrating responses and stewarding tasks.

Boundary conditions Review can be made more effective. in an experiment with 199 participants, Buçinca et al. [39] found that cognitive forcing functions, interventions that prompt people to think through a decision before accepting AI advice, significantly reduced overreliance compared with simple explanations. Participants, however, rated the most effective designs least favorably. The survey evidence of Lee et al. [5] also suggests that review is stronger where reviewers are confident in their own ability. The fallacy therefore concerns review that is assumed to work without being designed and resourced to do so.

Consequences A review requirement written into policy can provide formal assurance that the review does not deliver in practice. The effect is likely strongest where reviewers trust the system and have little time or domain expertise.

4.4. Fallacy 4: Perceived Productivity Equals Actual Productivity

Assumption How productive practitioners feel when using AI tools reflects how productive they are.

Why it persists. Visible output appears faster, and self-reports and satisfaction surveys are easy to collect at scale. The time spent prompting, waiting, reading and correcting is spread across the task and is harder to notice than the moment a draft appears.

Evidence. Becker et al. [4] ran a randomized controlled trial with 16 experienced open-source developers working on repositories they had contributed to for years; the repositories averaged more than 22,000 stars and over one million lines of code. Before the study, developers expected AI to speed them up by 24%. When AI tools were allowed, they took 19% longer to complete issues. Even after the study, they believed AI had sped them up by 20%. The authors note that the trial covers early-2025 tools in one setting.

Boundary conditions. Productivity gains are well documented: 15% more issues resolved per hour among customer-support agents [1], 40% less time on writing tasks [2], 55.8% faster completion of a defined programming task [18] and 26.08% more completed tasks in pooled field experiments with developers [19]. Gains tend to be largest for less experienced workers [1,19], while the most experienced customer-support agents saw small speed gains and small declines in quality [1]. The fallacy concerns the substitution of perception for measurement, which is most misleading for experts working in familiar, complex settings.

Consequences. Adoption and investment decisions based on perceived gains can misjudge the size of the effect and, in some settings, its direction. Because this fallacy currently rests on a single small trial, its evidence grade is limited, and replication is a priority (Section 5.3).

4.5. Fallacy 5: AI Removes Work

Assumption. Tasks delegated to AI disappear from the workload.

Why it persists. The visible production step, such as writing a draft or a function, takes much less time. The work that replaces it, such as specifying, checking and integrating, is distributed, less visible and rarely measured.

Evidence. Simkute et al. [15] identify four sources of productivity loss with generative AI: a shift in the user’s role from production to evaluation, unhelpful restructuring of workflows, interruptions, and automation making easy tasks easier and hard tasks harder. Bainbridge [12] made the parallel observation for industrial automation, where operators keep the tasks that could not be automated. Lee et al. [5] found that generative AI shifts critical thinking toward verification, integration and task stewardship. in the skill formation experiment of Shen and Tamkin [6], some participants spent substantial time composing queries and reviewing AI output, which offset the expected speed-up.

Boundary conditions. Where evaluation is cheap relative to production, the net saving can be large, as in the writing tasks studied by Noy and Zhang [2]. The fallacy concerns tasks where evaluation and integration are costly, which includes much expert work.

Consequences. Plans and staffing models that count only the production step underestimate the evaluation and integration work that remains. This fallacy is closely related to Fallacy 4, since unaccounted work is one mechanism behind the gap between perceived and actual productivity.

4.6. Fallacy 6: Accountability Moves To The Tool Or Vendor

Assumption. When an AI system makes an error, responsibility lies with the system or its provider.

Why it persists. The error originates in a system the user did not build, vendors present their tools as reliable, and conversational interfaces portray the system as an agent with its own voice. Air Canada’s defense in Moffatt v. Air Canada [9], discussed below, illustrates the reasoning.

Evidence. Courts and tribunals in three jurisdictions have rejected the assumption. In Moffatt v. Air Canada [9], the airline’s website chatbot told a customer that a bereavement fare could be claimed after travel, contrary to the airline’s policy page. Air Canada argued that it could not be held liable for information provided by its chatbot. The British Columbia Civil Resolution Tribunal rejected the argument, held the airline responsible for all information on its website, and found negligent misrepresentation. In Mata v. Avianca, Inc. [7], a United States federal court sanctioned the attorneys and their firm, holding them responsible for the accuracy of their filing regardless of the tool used. In Ayinde v London Borough of Haringey [8], the High Court of England and Wales held that lawyers who use AI have a professional duty to check the accuracy of its output against authoritative sources, and that those with leadership responsibilities in law firms and chambers must take practical measures to prevent misuse.

Boundary conditions. Some liability does move toward vendors. The EU’s revised Product Liability Directive treats software, explicitly including AI systems, as a product subject to no-fault liability for defects, and will apply to products placed on the market after 9 December 2026 [28]. Contracts may also allocate risk between customer and provider. Neither relieves professionals of their own duties toward clients and courts, as Ayinde v London Borough of Haringey [8] makes clear.

Consequences. Responsibility remains with the deploying organization and the professional who uses the output, whatever its source. Elish [40] describes a related risk: in automated systems, responsibility may be attributed to the nearest human operator even when that person had limited control over the system’s behavior, a position she calls a moral crumple zone. Professionals asked to sign off on AI output they cannot realistically check may find themselves in that position.

4.7. Fallacy 7: Skills and System Understanding Stay Intact When Work is Delegated

Assumption. Professionals retain their skills, and their understanding of the systems they are responsible for, while AI performs the work.

Why it persists. Skill loss is gradual and stays invisible as long as the AI keeps working. Delegating a task does not feel like forgetting how to do it, and the moment when the skill is needed again, a failure or an unavailable tool, is rare and unpredictable.

The evidence differs for its two components, skill erosion (7a) and the missing mental model of agent-built systems (7b), which are treated in turn.

4.7.1. Skill Erosion (7a)

Bainbridge [12] observed that when most of a task is automated, operators no longer practice the skills they need when automation fails. Shen and Tamkin [6] tested this for AI-assisted programming in a preregistered randomized experiment with 52 developers learning an unfamiliar Python library. Participants with access to a chat-based AI assistant scored 4.15 points lower on a 27-point quiz than the control group, which the authors report as a 17% score difference (Cohen’s $d=0.738$, $p=0.010$). The AI group was not significantly faster. The largest gap appeared in debugging questions; debugging is the skill most needed to supervise AI-generated code. The study measured outcomes within about one hour, and the authors call for longitudinal research.

Evidence from aviation points in the same direction. Casner et al. [35] asked 16 airline pilots to fly routine and non-routine scenarios in a Boeing 747-400 simulator. Instrument scanning and manual control skills were largely retained, but the cognitive skills of manual flying showed more frequent and significant problems. These problems were associated with how often pilots engaged in task-unrelated thought while the automation was flying.

Industry practice offers an illustration without measurement. in 2014, Toyota reintroduced manual workstations in some of its Japanese plants. As reported by SCDigest [34], drawing on Bloomberg News [24], the executive leading the program, Mitsuru Kawai, said the company needed to “sharpen our manual skills” and could not “simply depend on the machines that only repeat the same task over and over again”. No measurement of skill decline was reported. According to the same report, Toyota’s president attributed the loss of the earlier improvement mindset to rapid production growth rather than to automation. The case is therefore cited as an illustration only.

4.7.2. Missing Mental Model of Agent-Built Systems (7b)

Naur [36] argued that programming is primarily the programmers’ building of knowledge, a theory of the program, with documentation as an auxiliary and secondary product. On this view, a program decays when people modify it without grasping its underlying theory. A codebase produced largely by coding agents can exist without any person holding that theory.

Comprehension is already the dominant cost of software work. in a field study of 78 professional developers over 3,148 working hours, Xia et al. [37] found that developers spend about 58% of their time on program comprehension. That share is likely to grow for a team that must take over an agent-built codebase it never understood. Shen and Tamkin [6] note that their chat-based setting is likely a lower bound for cognitive offloading, since agentic coding tools require even less human participation.

The missing theory remains hidden while the agent is available, because the agent can re-read and modify the code. It becomes visible when access is lost or reduced (Fallacy 9) or when the agent fails on a task outside its frontier (Fallacy 2). No study was found that measures this situation directly. The argument in this subsection is an inference from the sources cited and is listed as an open question in Section 5.3.

4.7.3. Boundary Conditions and Consequences (7a and 7b)

Boundary conditions. Skill loss is not an inevitable consequence of AI use. In Shen and Tamkin [6], participants who used AI to build understanding, for example by asking conceptual questions or requesting explanations alongside generated code, reached average quiz scores of 65% to 86%, compared with 24% to 39% for those who delegated code generation or debugging. Casner et al. [35] similarly conclude that the retention of cognitive skills may depend on how actively pilots stay engaged in supervising the automation. Both findings point to cognitive engagement as the condition under which skills are preserved. For system understanding, the analogous condition would be that at least one person actively builds and maintains the theory of each agent-built system, a proposition that has not yet been tested.

Consequences. Organizations can end up responsible for work and systems that no one inside the organization can perform, maintain or explain without the tool.

4.8. Fallacy 8: Results Transfer From Benchmarks and Pilots

Assumption. Benchmark scores, vendor demonstrations and pilot results predict performance in the organization’s own work.

Why it persists. Benchmarks are quantitative, public and easy to compare, and pilots take place inside the organization. Pilots may also be run on tasks chosen because they are expected to succeed, and benchmark names suggest broader abilities than the benchmarks measure [10].

Evidence. Raji et al. [26] analyze widely used benchmarks and argue that construct validity is a generally neglected issue in their design, and that current benchmarking methods do not offer meaningful measures of general capabilities. Selbst et al. [17] describe a portability trap: a technical solution designed for one social context can be misleading, inaccurate or harmful when moved to another. in the field, Becker et al. [4] observed a 19% slowdown with tools that achieve strong benchmark scores, and Magesh et al. [25] found hallucination rates well above what providers of legal research tools had claimed.

Boundary conditions. Results transfer better when the evaluation tasks resemble the work. The consulting experiment of Dell’Acqua et al. [3], conducted with the firm on realistic consulting tasks, showed that results depended on whether a task lay inside or outside the frontier, which only an evaluation on representative tasks can reveal. A pilot that samples the organization’s real task mix, including difficult and atypical cases, is the most direct test available.

Consequences. A pilot limited to tasks inside the frontier will not reveal the failures that production tasks outside it can produce (Fallacy 2). Because much of the evidence for this fallacy is conceptual, its grade is limited.

4.9. Fallacy 9: The Model is Available

Assumption. Access to a given model or provider will continue on current terms, so workflows, staffing and deadlines need no fallback.

Why it persists. Major providers are large, well-resourced companies, and outages in daily use tend to be short. Outside regulated sectors, dependence on a single AI provider may not be assessed as a concentration risk.

Evidence Access to AI services has been lost or ended through at least four routes:

• Technical outage. OpenAI’s post-mortem reports that all its services, including the API and ChatGPT, were significantly degraded or unavailable for about four hours on 11 December 2024 [29].
• Regulatory action. On 30 March 2023, the Italian data protection authority ordered an urgent temporary limitation of OpenAI’s processing of personal data of users in Italy [23]. OpenAI then blocked access to ChatGPT in Italy.
• Export controls. On 12 June 2026, Anthropic suspended access to two of its models, Claude Fable 5 and Claude Mythos 5, after a United States government export control directive took effect immediately [31]. The controls were lifted on 30 June 2026, and access was restored from 1 July 2026 [32].
• Scheduled retirement. Providers retire models on a regular basis. OpenAI’s documentation states that generally available models receive at least six months’ notice before shutdown unless safety or compliance concerns require a faster timeline; text-davinci-003, for example, was shut down on 4 January 2024 [30]. Anthropic commits to at least 60 days’ notice for publicly released models and retired Claude 3 Opus on 5 January 2026 [33].

The EU Digital Operational Resilience Act, Regulation (EU) 2022/2554 [27], shows how one sector already treats comparable dependencies. It requires financial entities to assess concentration risk in ICT third-party services, including the availability of alternative providers, and to maintain exit strategies that ensure business continuity.

Boundary conditions. The documented outage lasted hours, and scheduled retirements come with notice periods, so their effect can be planned for where model dependencies are tracked. The fallacy matters most where no fallback exists, where retirements are not monitored, and for abrupt losses such as regulatory or export-control measures, which come with little or no warning.

Consequences. Work stops or degrades when access is lost. The impact is larger where skills and system understanding have eroded (Fallacy 7).

5. Discussion

5.1. Interactions Between Fallacies

The fallacies have distinct mechanisms and can compound in practice. Table 3 lists the combinations supported or suggested by the reviewed evidence.

The combination of Fallacies 7 and 9 deserves particular attention. Consider a team that uses a coding agent to build and extend an internal system over several months. Reviews check whether changes work, and no team member builds a full understanding of the design. Access to the agent is then lost through an outage, a regulatory measure, an export control or the retirement of the model the workflow depends on, each of which has occurred (Section 4.9). The team must now correct a production defect in a codebase whose rationale no one holds. Under normal conditions comprehension already takes most developer time [37], and debugging skills, which are needed here, were the most reduced in the one experiment that measured them [6]. This scenario is constructed from the cited evidence and has not been studied directly.

5.2. Implications For Practice

The fallacies suggest measures at three levels. Measures with direct support in the reviewed sources are cited; the others are proposals that require evaluation.

Individual practitioners can verify AI output against sources independent of the system that produced it, in proportion to the stakes (Fallacy 1). They can track task types on which the tool has failed them (Fallacy 2). When working in areas they are responsible for, they can use AI to build understanding, for example by asking for explanations, a pattern associated with retained mastery [6] (Fallacy 7).

Team and engineering leads can budget evaluation and integration time explicitly (Fallacy 5), and can apply the design approaches proposed by Simkute et al. [15], namely continuous feedback, system personalization, ecological interface design, task stabilization and clear task allocation. They can design review steps with cognitive forcing in mind, accepting that the most effective designs may be the least popular [39] (Fallacy 3). They can measure productivity through outcomes such as cycle time, rework and defect rates, alongside self-reports (Fallacy 4). They can require that someone can explain the design of each agent-built system the team owns, and schedule periodic unassisted work on critical tasks (Fallacy 7).

Governance, risk and procurement functions can assign named owners to AI-assisted outputs that leave the organization, and can follow the expectation of Ayinde v London Borough of Haringey [8] that leadership takes practical measures against misuse (Fallacy 6). They can require pilots on the organization’s own tasks, including difficult cases (Fallacy 8). and they can treat AI providers as critical third parties: inventory dependencies, monitor announced retirements, maintain tested fallbacks and degraded-mode procedures, and exercise exit plans, following the logic of the EU Digital Operational Resilience Act [27] (Fallacy 9).

5.3. Research Agenda

The review identifies seven gaps:

1.

Maintenance after loss of access. No study was found on how teams maintain agent-built software after losing access to the agent (Fallacies 7 and 9). A controlled study comparing teams that build a system with or without agents and then maintain it without AI would test the inference in Section 4.7.2.

2.

Perception and measurement. The gap between perceived and measured productivity rests on a single trial with 16 developers [4]. Replications with larger samples and in other professions are needed (Fallacy 4).

3.

Long-term skill effects. The only experiment found measures skill within about one hour [6]. Longitudinal studies over months are needed (Fallacy 7).

4.

Further forms of reduced access. The frequency and impact of outages, rate limits, price changes and retirements on professional work have not been measured (Fallacy 9).

5.

Generalization. Key studies cover single domains, such as consulting [3] and open-source software [4], and specific tool generations. Replication across domains and over time is needed (Fallacy 8).

6.

Measuring review effectiveness. Practical methods are needed to measure how well human review catches AI errors in daily work (Fallacy 3).

7.

Validation of the framework. Practitioner surveys or expert panels could test whether the nine fallacies are recognized, distinct and complete, whether the evidence grades are judged appropriate, and whether the self-assessment instrument (Appendix A) is reliable and valid.

5.4. Limitations

The review was not systematic, and relevant work in subscription databases or other languages may have been missed. Several studies are small or cover a single domain, and AI tools change faster than research is published. Five sources are preprints or working papers (Table 1), including the principal evidence for Fallacies 2 and 4 [3,4]. The case law comes from three common-law jurisdictions. Four sources were authored or published by Anthropic: the skill formation study [6], two statements on suspended and restored access [31,32] and the model deprecation documentation [33]; the AI tool used in preparing the paper is also an Anthropic model. Two further sources are OpenAI documentation [29,30], and two studies come from researchers at Microsoft Research [5,15]. Web search engines do not support fully reproducible queries, so the search is documented by its strategy and strands (Section 3.3) rather than by exact query strings and hit counts. The evidence grades are the author’s assessment and were not independently rated. The number of fallacies is a design choice. Fallacies 4 and 5 could be merged, giving a shorter list, at the cost of separating measurement from mechanism less clearly.

6. Conclusions

This paper proposed nine fallacies of professional AI usage, grounded in human factors research, controlled studies, court decisions, provider documentation and regulation. Each fallacy names an assumption that appears reasonable when AI tools are introduced and that the evidence shows to be false under identifiable conditions of professional use. The boundary conditions are an integral part of each fallacy: in one study, AI tools slowed down experienced developers in familiar settings, while in others they sped up less experienced workers on well-defined tasks.

The analysis of interactions suggests that the fallacies can compound. in particular, systems built largely by coding agents may remain maintainable only for as long as the agent is available, a risk that research has not yet measured.

The list is intended as a shared vocabulary for practitioners, managers and governance functions, and as a starting point for empirical validation. Like the distributed computing list that inspired its format, it is meant to help organizations check their own assumptions before those assumptions fail in practice.