Are AI Detectors Accurate? What the Comparison Tests Actually Measured

Ask an AI detector’s marketing page whether it is accurate and you get a single confident number — “99% accuracy,” “less than 1% false positives.” Ask the peer-reviewed comparison tests the same question and you get something far more uncomfortable: it depends entirely on whose text you feed in. These tools are good at one thing and bad at another, and the split is not random. They clear human writing more reliably than they catch machine writing — which means the mistakes they do make land almost entirely on people, in exactly the situations where being wrong ruins someone’s week. That asymmetry, not any headline accuracy figure, is the whole story of whether AI detectors can be trusted.

Two numbers make the point before we go any further. In one 2026 study of academic abstracts, the detector ZeroGPT correctly identified AI-written text with 100% accuracy but recognized genuine human writing only 2.08% of the time; its competitor GPTZero scored a flat 0% on the human samples (Shamsi et al., 2026, p. 130). Read that twice. The tools were nearly flawless at flagging machines and nearly useless at clearing humans. A detector that labels 98% of real human work as “AI” is not a detector — it is a random accusation generator with a good user interface.

The one asymmetry that matters

The most careful survey of the field comes from Debora Weber-Wulff and a large European team, who ran 54 test cases through the major detectors — 756 individual tests in all. Their finding is the sentence to memorize: human-written texts “are usually identified quite accurately (accuracy above 80%),” but the tools’ “ability to identify AI-generated text is under question as their accuracy in many studies was only around 50% or slightly above” (Weber-Wulff et al., 2023, p. 8). One earlier evaluation they cite went further, concluding the tools were “no better than random classifiers” at the core task of spotting machine text (Weber-Wulff et al., 2023, p. 7).

A separate 2025 literature review reaches the same conclusion from a different pile of studies. Summarizing the work of Orenstrakh and colleagues, it reports that detectors “produce more accurate findings for human-made input than ChatGPT-created input, yet they are still unreliable for use in the academic setting” (Gotoman et al., 2025, p. 5). The same review notes one tool misidentified up to 8% of genuine abstracts as AI-generated, and that in a related test people could tell a machine-written news story from a human one only 52% of the time — a coin flip (Gotoman et al., 2025, p. 5).

Put those together and the “are they accurate” question splits in two. Are they accurate at confirming a human wrote something? Reasonably — usually above 80%. Are they accurate at catching AI, the thing they are sold to do? Barely — often around 50%. The product does its advertised job worst.

The clearest admission of this came from the company best positioned to know. When OpenAI released its own AI Text Classifier in early 2023, it disclosed that the tool correctly flagged just 26% of AI-written text while wrongly flagging 9% of human text as machine-made (OpenAI, 2023). Six months later OpenAI quietly killed the product, citing its “low rate of accuracy.” The maker of ChatGPT could not reliably detect ChatGPT.

Why the errors land where they hurt

A 50% miss rate on AI text is embarrassing, but it is a safe failure — a cheater slips through, and no innocent person is harmed. The dangerous failure is the other one: flagging a real human as a machine. And the comparison tests are unanimous that these false positives are not spread evenly. They concentrate on specific kinds of writers.

The landmark result belongs to Weixin Liang and colleagues at Stanford. They ran seven widely used GPT detectors over two sets of essays: writing by US eighth-graders, and TOEFL essays by non-native English speakers. On the American students, the detectors were near-perfect. On the non-native essays, they posted an average false positive rate of 61.22% — and 97.80% of those TOEFL essays were flagged as AI by at least one detector (Liang et al., 2023, p. 2). The mechanism is not mysterious. Detectors lean on “perplexity,” a measure of how statistically surprising word choices are, and second-language writers naturally use a narrower, more predictable vocabulary. To prove the point, the researchers prompted GPT-4 to “simplify word choices as if written by a non-native speaker,” and the misclassification rate for genuine US essays jumped from 5.19% to 56.65% (Liang et al., 2023, p. 2). Write in plainer English and a detector starts calling you a robot. This is the direct link between detection and the penalty on non-native English writers: the tools mistake a second language for a language model.

The Persian-abstract study named above shows the same collapse in another population, and Ahmed Elkhatat’s team found detectors throwing false positives and “uncertain” verdicts on human-written samples while systematically letting GPT-4 text pass as human (Elkhatat et al., 2023, p. 9). Same pattern, three research groups, three languages: the accuracy the vendors quote is an average that dissolves the moment you look at who is standing under it. It is precisely the fluent-but-wrong confidence that makes these outputs so dangerous — the same failure mode behind fabricated citations, where a tool produces a clean, plausible verdict with nothing real underneath it.

Paraphrasing quietly breaks the whole thing

Even the modest accuracy detectors do have is fragile. Vinu Sadasivan and colleagues showed that running AI text through an ordinary paraphraser drops detector accuracy from 97% to 80% with a light rewrite, and to 57% with a slightly heavier one — without meaningfully degrading the text (Kumar et al., 2024, p. 6). Worse, they proved a mathematical ceiling: as language models get better at imitating human writing, the statistical gap a detector needs shrinks, and “the performance of even the best possible detector decreases” while the false positive rate climbs (Kumar et al., 2024, p. 11). Detection does not get easier as models improve. It gets structurally harder.

Their most alarming demonstration targeted the “retrieval” detectors that store known AI outputs and match against them. By feeding paraphrased human essays into such a database, an attacker made the tool falsely classify all 100 human passages as AI-generated (Kumar et al., 2024, p. 19). The failure runs in both directions: a student who paraphrases AI text sails through, and a student whose honest work resembles something in the database gets convicted.

Where the harm actually happens

This is not a lab curiosity. AI detectors have moved into classrooms and courtrooms, and the false positives now come with consequences. A 2026 study on “automation bias” opens with the problem in one line: “A single percentage in an artificial intelligence detection report can change how a teacher reads a student paper” (Du et al., 2026, p. 2). Its experiments found that strong numerical warnings paired with red visual risk cues pushed teachers toward guilty verdicts even against their own judgment — so the authors urge universities to stop treating detection reports as “decisive evidence” and to add independent review, appeals, and human oversight before any accusation (Du et al., 2026, p. 14).

Institutions are already retreating. Vanderbilt University disabled Turnitin’s AI detector in 2023, noting that of the 75,000 papers it submitted the prior year, roughly 750 could have been wrongly flagged if the tool had been running (Vanderbilt University, 2023). More than fifty universities across the US, UK, Canada and Australia — including Yale, Johns Hopkins and Waterloo — have since disabled or restricted the same feature. Turnitin launched claiming a false positive rate under 1%, then quietly revised it upward to around 4% at the sentence level.

And the lawsuits have started. A Yale student sued in 2025 after being flagged by GPTZero, alleging the tool is unreliable and biased. A University of Michigan undergraduate sued in 2026 over an accusation her filing says rested on “subjective judgments” about her writing style — one instructor reportedly admitted grading had made him “paranoid and inclined to see AI everywhere,” then filed the complaint anyway (Sourwine, 2026). A Palo Alto family filed a federal civil-rights suit over their son’s punishment. Every one of these cases is a false positive with a name attached.

So — are AI detectors accurate?

The honest answer has three parts. At confirming that a fluent native English speaker wrote something ordinary, they are fairly reliable. At catching actual AI text — their entire reason to exist — they hover near a coin flip and fall apart under a five-minute paraphrase. And at the one job where a mistake destroys something, flagging a real person as a machine, they fail hardest against the writers least equipped to fight back: non-native speakers, students with plain prose, anyone whose sentences happen to be statistically calm.

That is why a detector score should never be evidence on its own. It is a prompt to look closer — at draft history, at version logs, at a conversation with the writer — not a verdict. Weber-Wulff’s team, Elkhatat’s team, Liang’s team, and the OpenAI engineers who shut down their own tool all reached the same place from different directions: these numbers describe a probability, not a fact, and the probability is worst exactly where the cost of being wrong is highest. Treat any percentage from a detector the way you would treat an anonymous accusation — as something that has to be checked, never as something already proven.