Where this comes from

In a preregistered online experiment published in Science (Noy and Zhang, July 2023), 453 college-educated professionals did occupation-specific writing tasks, and half were randomly given ChatGPT. "The average time taken decreased by 40% and output quality rose by 18%." Inequality between workers decreased.

Only the abstract was read. These were set tasks in an experiment, with an early version of ChatGPT, so real jobs may differ.

Strength: Strong for what it measured · July 2023

A study of 5,179 customer support agents (Brynjolfsson, Li and Raymond; published in the Quarterly Journal of Economics, 2025) found an AI assistant raised issues resolved per hour by 14% on average, "including a 34% improvement for novice and low-skilled workers but with minimal impact on experienced and highly skilled workers."

Only the abstract page was read. It covers one kind of work, and the tool was introduced in stages rather than in a lab trial.

Strength: Strong for this setting · 2025

In a randomised trial at college level (Kestin et al., 2025; full article), students "learn significantly more in less time" with an AI tutor than in an in-class active learning lesson, and felt more engaged and motivated. The tutor was custom-built on the same teaching practices as the class.

Only the abstract was read, so effect sizes and sample size are not given here. This was a purpose-built tutor, not a general chatbot.

Strength: Moderate · June 2025

A study with more than 700 Boston Consulting Group consultants (Dell'Acqua et al.), as summarised by MIT Sloan, found GPT-4 could improve performance by nearly 40% on tasks within its abilities. On a task outside them, performance dropped by an average of 19 percentage points.

This rests on MIT Sloan's summary because the paper's page refused access. The summary's 19-point average is not reconciled with its separate 13 and 24 point figures.

Strength: Moderate until the paper is read · October 2023

In a randomized trial by METR (published 10 July 2025), 16 experienced open-source developers worked on 246 real issues, with AI tools randomly allowed or not. When AI was allowed, tasks took 19% longer. The developers had expected a 24% speedup and, afterwards, still believed it was about 20%.

The authors list limits: a small sample, one setting, early-2025 tools and possible sampling bias. They say they do not claim AI slows most developers. METR's page now carries a notice that the results are out of date.

Strength: Strong for what it measured, weak for generalising · 10 July 2025

METR's update of 24 February 2026 used late-2025 data. Returning developers showed about 18% less time with AI and new developers about 4% less, but the confidence intervals include zero. METR says its data is "only very weak evidence" for the size of the increase: many developers refused to work without AI or withheld tasks, and self-reported speedups are unreliable. The experiment is being redesigned.

Its value is showing how hard the measurement is. Secondary blogs report this update in conflicting ways, so this guide uses only METR's own page.

Strength: Moderate to weak · 24 February 2026

In an Anthropic randomized trial (29 January 2026), 52 mostly junior engineers learned the Trio library. Completion time was not significantly different (about 2 minutes faster with AI, p = 0.391). On a later quiz the AI group scored 50% and the no-AI group 67% (p = 0.01), with the largest gap on debugging. Lower-scoring usage patterns delegated code or used AI only to debug; patterns averaging 65% or higher asked conceptual or follow-up questions.

Anthropic is an AI vendor studying its own product category, although the result is unflattering. The authors say the pattern analysis is correlational, the sample small, comprehension was measured shortly afterwards, and the tool was a chat assistant, not an agentic one.

Strength: Moderate (small RCT) · 29 January 2026

A field experiment with nearly 1,000 high-school maths students (Bastani et al., PNAS abstract; working paper) found GPT-4 access raised practice grades by 48% with a standard chat interface and 127% with a tutor version built to safeguard learning. When access was removed, the standard group did 17% worse than students who never had it. The authors say the safeguards largely mitigated this.

The abstract was read, not the full paper. Peer-reviewed. That it was one school in Turkey comes from search summaries.

Strength: Moderate

The Stack Overflow Developer Survey 2025 reports that 47.1% of all respondents (50.6% of professionals) use AI tools daily. 46% distrust the accuracy of AI output against 33% who trust it, and 3.1% highly trust it. The top frustration, at 66%, is "AI solutions that are almost right, but not quite".

A large, self-selected, self-reported survey. It measures attitudes, not outcomes. Fieldwork dates are not on the page.

Strength: Moderate

The DORA 2025 report describes AI's primary role as "an amplifier, magnifying an organization's existing strengths and weaknesses." A Google blog summary reportedly adds that adoption is linked to higher throughput but also to instability unless testing and quality assurance keep pace.

Large survey, correlational, from a Google-run research programme. Only the quoted sentence was confirmed on the page; the throughput and instability point rests on a search summary. Exact numbers are deliberately not given.

Strength: Moderate

The Anthropic Economic Index (15 January 2026) says computer and maths tasks, mostly coding, are "a third of conversations on Claude.ai and nearly half of 1P API traffic". The Claude.ai share fell from 40% (March 2025) to 34% (November 2025). The report says "none of the measures should be taken as exact or definitive."

Data about one vendor's users only. It is not a measure of AI use in general.

Strength: Moderate · 15 January 2026

Anthropic's prompting guide says "Claude responds well to clear, explicit instructions" and suggests a test: "Show your prompt to a colleague with minimal context on the task and ask them to follow it. If they'd be confused, Claude will be too." It also says examples "are one of the most reliable ways to steer Claude's output format, tone, and structure."

Official documentation from the company that makes the tool. It describes how the tool works and does not measure results. Details can change, so check the page.

Strength: Official guidance · Checked 1 November 2026

The Claude Code model docs describe haiku as for "simple tasks", sonnet for "daily coding tasks", opus for "complex reasoning tasks" and fable for "your hardest and longest-running tasks". /model switches during a session and saves your choice as the default unless you pick session-only. Its costs page says "Sonnet handles most coding tasks well and costs less than Opus."

Official documentation from the company that makes the tool. It describes how the tool works and does not measure results. Details can change, so check the page.

Strength: Official guidance · Checked 1 November 2026

The Claude Code costs page says "Token costs scale with context size." It advises /clear when switching to unrelated work because "Stale context wastes tokens on every subsequent message", and warns that "compacting a large context is itself a large request." /usage shows token use for the session.

Official documentation from the company that makes the tool. It describes how the tool works and does not measure results. Details can change, so check the page.

Strength: Official guidance · Checked 1 November 2026

The Claude Code permissions page says that in plan mode Claude "reads files and runs read-only shell commands to explore but doesn't edit your source files". The costs page says to "Press Shift+Tab to cycle to plan mode before implementation." The best practices page suggests having Claude interview you, writing a spec, then building in a fresh session.

Official documentation from the company that makes the tool. It describes how the tool works and does not measure results. Details can change, so check the page.

Strength: Official guidance · Checked 1 November 2026

The Claude Code best practices page advises: "Give Claude something it can run that comes back pass or fail." It also says to ask for "evidence: the command it ran and its output, or the screenshot."

Official documentation from the company that makes the tool. It describes how the tool works and does not measure results. Details can change, so check the page.

Strength: Official guidance · Checked 1 November 2026

The Claude Code skills docs say a skill is a SKILL.md file of instructions, kept in .claude/skills/<name>/ for a project or ~/.claude/skills/<name>/ for you, and run with /skill-name or loaded when relevant. "Unlike CLAUDE.md content, a skill's body loads only when it's used, so long reference material costs almost nothing until you need it." Setting disable-model-invocation: true "prevents Claude from invoking the skill on its own; you can still run it directly."

Official documentation from the company that makes the tool. It describes how the tool works and does not measure results. Details can change, so check the page.

Strength: Official guidance · Checked 1 November 2026

The Claude Code memory docs say CLAUDE.md files give Claude persistent instructions that it reads at the start of every session, and that /init generates a starting file. They advise: "target under 200 lines per CLAUDE.md file." They also warn: "Claude treats them as context, not enforced configuration."

Official documentation from the company that makes the tool. It describes how the tool works and does not measure results. Details can change, so check the page.

Strength: Official guidance · Checked 1 November 2026

The Claude Code subagent docs say each subagent runs in its own context window and "sends its own requests, which count toward the same usage limits as your main conversation." The best practices page says they pay off for parallel work, keeping long output out of the main conversation and getting a reviewer who has not seen how the code was written. By default, it says, you get "parallelism or a fresh pair of eyes out of it, but not a smarter answer."

Official documentation from the company that makes the tool. It describes how the tool works and does not measure results. Details can change, so check the page.

Strength: Official guidance · Checked 1 November 2026

The Claude Code permissions page says "Permission rules are enforced by Claude Code, not by the model", and are evaluated in the order deny, ask, allow. On bypass mode it says: "Only use this mode in isolated environments like containers or VMs where Claude Code can't cause damage."

Official documentation from the company that makes the tool. It describes how the tool works and does not measure results. Details can change, so check the page.

Strength: Official guidance · Checked 1 November 2026

The Claude Code MCP docs describe MCP as "an open source standard for AI-tool integrations" that gives Claude Code access to tools, databases and APIs. They warn: "Verify you trust each server before connecting it. Servers that fetch external content can expose you to prompt injection risk."

Official documentation from the company that makes the tool. It describes how the tool works and does not measure results. Details can change, so check the page.

Strength: Official guidance · Checked 1 November 2026

The Claude Code costs page says "The Session block at the top of /usage shows detailed token usage statistics for your current session", and that on a paid plan "/usage also shows a breakdown of what counts against your plan limits, with recommendations". It adds: "Claude Pro, Max, and Ultra subscribers have usage included in their subscription, so the session cost figure isn't relevant for billing purposes." It says "Run /context to see what's consuming space", and "Usage credits let you keep working past your plan's usage limit."

Official documentation from the company that makes the tool. It describes how the tool works and does not measure results. Details can change, so check the page.

Strength: Official guidance · Checked 9 November 2026

The Claude Code status line docs say that for subscribers "the rate_limits object contains a rolling five_hour window and a weekly seven_day window. Each window provides used_percentage, from 0 to 100, and resets_at, the Unix epoch seconds when the window resets."

Read through documentation excerpts rather than the whole page. Official documentation from the company that makes the tool. It describes how the tool works and does not measure results. Details can change, so check the page.

Strength: Official guidance · Checked 9 November 2026

The Claude Code settings reference says cleanupPeriodDays sets "how many days Claude Code keeps session transcripts and other application data before deleting them." The default is 30, the sweep "deletes transcripts without showing a message", and "Setting 0 fails validation, so pick a large value such as 3650 for long retention."

Read through documentation excerpts rather than the whole page. Official documentation from the company that makes the tool. It describes how the tool works and does not measure results. Details can change, so check the page.

Strength: Official guidance · Checked 9 November 2026

The Claude Code memory docs say "CLAUDE.md files can import additional files using @path/to/import syntax. Imported files are expanded and loaded into context alongside the CLAUDE.md that references them." They list ~/.claude/CLAUDE.md as user instructions, for "Personal preferences for all projects". For folders added with --add-dir, "By default, CLAUDE.md files from these directories are not loaded."

The first two thirds of the page were read in full, and the table of locations through an excerpt. Official documentation from the company that makes the tool. It describes how the tool works and does not measure results. Details can change, so check the page.

Strength: Official guidance · Checked 9 November 2026

In a randomised experiment (Peng et al., February 2023), professional programmers recruited through Upwork, using GitHub Copilot, finished an HTTP server task in JavaScript 55.8% faster than a control group (95% confidence interval 21 to 89%). 95 were recruited and 35 from both groups completed it. Less experienced developers benefited more.

Authors include GitHub and Microsoft Research staff, and both sell Copilot. It is one standard task, and the authors say it "does not examine the effects of AI on code quality".

Strength: Moderate · February 2023

The Veracode 2025 GenAI Code Security Report (30 July 2025) tested 100+ language models on 80 coding tasks built from known weakness types (CWEs). In 45% of cases the models introduced a flaw of an OWASP Top 10 class. Java failed in over 70% of cases, Python, C# and JavaScript between 38% and 45%, cross-site scripting (CWE-80) in 86% and log injection (CWE-117) in 88%. The report says models are "getting better at coding accurately but are not improving at security".

Veracode sells code-scanning and fixing tools. This is a synthetic benchmark in which a secure and an insecure option both existed. It does not show that 45% of real AI-written code is vulnerable, although some outlets state it that way.

Strength: Moderate · 30 July 2025

Refero Styles, marked as a beta, says "Browse 2,000+ AI-readable design systems from leading product websites." Each one, it says, comes with "colors, typography, spacing, components, and a DESIGN.md you can use in Cursor, Claude Code, Codex, v0, or Lovable."

The site describing its own product. Its pricing and terms were not checked.

Strength: Product description · Checked 9 November 2026

The UK Government AI Playbook (10 February 2025) says AI systems "are also not guaranteed to be accurate", that you should "minimise the risk of privacy intrusion from the outset", and that "humans validate any high-risk decisions influenced by AI". It names prompt injection and hallucinations among threats specific to AI.

Guidance written for UK government work, not a study. The principles were read, not every section. Its acknowledgements list industry contributors including Amazon, Google, IBM and Microsoft.

Strength: Strong as official guidance · 10 February 2025

The OWASP Top 10 for LLM Applications 2025 lists: prompt injection; sensitive information disclosure; supply chain; data and model poisoning; improper output handling; excessive agency; system prompt leakage; vector and embedding weaknesses; misinformation; unbounded consumption.

It ranks risks by expert judgement. It is not measured incidence.

Strength: Strong as a community-consensus risk list

The UK NCSC (8 December 2025) argues that prompt injection is not SQL injection: language models have no inherent boundary between instructions and data, so prompt injection "may never be totally mitigated". It advises reducing impact through least privilege, deterministic non-LLM safeguards and logging, being wary of products that claim to "stop" it, and accepting that if the risk is intolerable, an LLM may not suit the use case.

Analysis, not an empirical study.

Strength: Strong as authoritative government technical opinion · 8 December 2025

The Guidelines for secure AI system development (v1.0, 27 November 2023) cover four areas: secure design, development, deployment, and operation and maintenance. They align with NIST's secure software development framework and CISA's secure-by-design approach.

Guidance, not outcome evidence. The page did not list partner agencies, so none are claimed here.

Strength: Strong as guidance · 27 November 2023

NIST's Generative AI Profile (AI 600-1), published 26 July 2024, is a "cross-sectoral profile of and companion resource for the AI Risk Management Framework (AI RMF 1.0) for Generative AI", "intended for voluntary use".

Voluntary, not a certification. The page was only partly read, and the current status of the executive order it was issued under was not checked, so nothing is claimed about it.

Strength: Strong as a reference · 26 July 2024

An unofficial EU AI Act implementation tracker (last updated 31 August 2026) lists these dates: prohibitions and the AI-literacy duty from 2 February 2025; general-purpose AI obligations from 2 August 2025; Article 50 transparency from 2 August 2026; high-risk Annex III systems from 2 December 2027; Annex I systems from 2 August 2028.

Unresolved: sources disagree on whether the "Digital Omnibus" amendment that would delay some dates is in force or still a draft. Do not rely on these dates. Check the EU Official Journal (EUR-Lex). The tracker is not an official source.

Strength: Moderate until checked against the Official Journal · Tracker last updated 31 August 2026

In 2023, Samsung staff reportedly pasted internal source code into ChatGPT, and the company temporarily restricted generative AI on company devices (TechRadar; CSO Online).

The original report was not read, and the company memo did not say what code leaked.

Strength: Moderate (consistent press reports) · 2023

EchoLeak (CVE-2025-32711) was reported as a zero-click indirect prompt injection flaw in Microsoft 365 Copilot, found by Aim Security, rated CVSS 9.3 and patched, with no evidence of exploitation in the wild (The Hacker News).

One aggregator gave a conflicting patch date. Check the National Vulnerability Database or the vendor's advisory before relying on details.

Strength: Moderate

Authors sued Anthropic in Bartz v. Anthropic over pirated books. A ruling in June 2025 reportedly found training itself to be fair use but not the pirated library, and a US$1.5 billion settlement was reportedly given final approval in July 2026 (Jurist).

Disclosure: the defendant makes Claude, a product this guide also cites as a source. Dates and details vary slightly between sources.

Strength: Moderate

In a randomised lab trial (Bono and Xu, Microsoft, October 2024), 181 participants recruited through Upwork completed IT admin tasks in a simulated environment, with half given Security Copilot. "Copilot subjects experienced a 34.53% improvement in overall accuracy and a 29.79% reduction in task completion time." Multiple-choice questions took 25.13% longer with it.

Written by Microsoft staff about a Microsoft product, in a simulated lab rather than live work. The discussion section was not read in full.

Strength: Moderate, vendor-run · October 2024

Anthropic reported (13 November 2025) that a group it assesses "with high confidence" as Chinese state-sponsored used Claude Code against roughly thirty targets and "succeeded in a small number of cases". Anthropic says the AI did "80-90% of the campaign", with humans at about four to six decision points, and that Claude "occasionally hallucinated credentials", "an obstacle to fully autonomous cyberattacks".

Vendor self-report, not independently verified. Words such as "first" and the 80-90% figure are Anthropic's framing, and Anthropic says it only sees its own product's usage. No independent rebuttal was retrieved.

Strength: Weak to moderate · 13 November 2025

Prompt injection is treated as an unsolved problem by both the NCSC and OWASP (LLM01). EchoLeak is a reported real-world example.

Strength: Strong consensus

Two publicly reported CEO memos from 2025 set out expectations. One, from Shopify (April 2025), reportedly made "reflexive AI usage" a baseline expectation, put AI use into performance reviews, and asked teams to show AI cannot do a job before requesting headcount. The other, from Duolingo (28 April 2025), announced an "AI-first" approach.

The memos themselves were not read; these are press reports of two companies' stated intent. They are examples, not a representative sample, and are not an endorsement of either company.

Strength: Weak to moderate · April 2025

Employment-law commentary notes that there is no shared definition of "AI literacy" in hiring. Statistics on how many job adverts mention AI skills exist from several analytics firms, but they vary by method.

No primary advert-analysis report was opened, so this guide reproduces no advert percentages.

Strength: Weak until primary reports are checked

The World Economic Forum Future of Jobs Report 2025 is reported to survey about 1,000 employers covering 14 million workers in 22 industries and 55 economies. According to summaries, AI and big data, networks and cybersecurity, and technological literacy are among the fastest-growing skills to 2030, alongside creative thinking, resilience and flexibility.

The report page could not be read (access refused), so this rests on summaries. These are employer expectations, not outcomes.

Strength: Moderate once verified

The Stanford Digital Economy Lab paper Canaries in the Coal Mine (Brynjolfsson, Chandar and Chen; revised 12 August 2026) uses ADP payroll data to June 2026. It finds employment of 22-to-25-year-olds in AI-exposed occupations "now stands 19% below where it would be had it kept pace with that of their less-exposed peers."

The authors call these "early, descriptive indicators... rather than causal estimates". The effect weakens with education controls and some trends predate generative AI. The paper has been revised, so the number can move between revisions. No rebuttal was retrieved.

Strength: Moderate, descriptive · Revised 12 August 2026

A study linking adoption surveys to Danish labour records (Humlum and Vestergaard, NBER, revised March 2026) estimates "precise null effects on earnings and recorded hours", "ruling out effects larger than 2% two years after the launch of ChatGPT". Workers still report productivity benefits, and adopters move into higher-paying occupations, "though still too few to move average earnings."

Only the abstract page was read. It covers one country and the early period after ChatGPT launched. It sits awkwardly beside the task-level gains above, which is why it is here.

Strength: Moderate to strong for Denmark · Revised March 2026

A statistic attributed to an MIT report on the failure rate of generative AI pilots was widely reported and has been criticised for thin methodology, a narrow definition of "failure" (no measurable profit-and-loss impact within about six months) and an author tie to a related technology.

This guide does not use the figure as evidence. It is listed here because readers will meet it.

Strength: Weak

Two trials in NEJM AI (late 2025) tested ambient AI scribes for clinical notes. A UCLA trial (Lukac et al.) of 238 physicians compared two products with usual care and found one reduced time spent on notes, with modest burnout improvements and occasional clinically significant note inaccuracies. A pragmatic trial (Afshar et al.) of 66 practitioners reported work exhaustion down 0.44 on a 5-point scale (PubMed; News-Medical summary).

Small trials measuring time and burnout, not patient outcomes. Funding and vendor ties were not checked, citation details vary, and the papers were not read.

Strength: Moderate, small · Late 2025

In Mata v. Avianca (S.D.N.Y.), a sanctions order of 22 June 2023 by Judge Castel concerned a lawyer who used ChatGPT and cited fabricated cases (overview; order).

The court order itself could not be opened (access refused), so details such as the sanction amount are not given.

Strength: Strong that it happened · 22 June 2023

Damien Charlotin's AI Hallucination Cases database listed "2241 cases identified so far" (updated 29 October 2026): 1,294 involving self-represented litigants, 884 lawyers and 33 judges. By country: USA 1,514, Canada 238, Australia 112, UK 89.

It counts court decisions discussing hallucinated content, some only alleged, and "does not track the (necessarily wider) universe of all fake citations". It is curated by one researcher and is not a rate per filing.

Strength: Strong that it happens and is rising; not a rate · Updated 29 October 2026

A Bank of England and FCA survey (21 November 2024, 118 respondents) reportedly found 75% of firms using AI and a further 10% planning to within three years. The FCA says it has no AI-specific rules and supervises under existing frameworks, and FINRA Regulatory Notice 24-09 (June 2024) says existing supervision and communications rules apply to AI-generated content (law-firm summary).

Regulatory position only: no outcome study for finance was found, no 2025 survey was found, and no primary document was read.

Strength: Weak to moderate · 21 November 2024