The popular version is simple: paste the essay, read the percent, trust the red flag.
It is not that simple.
An AI detector is closer to a smoke alarm than a courtroom. Useful. Also capable of screaming when nothing is burning - or staying quiet when there is smoke.
Research contests show the same pattern: ace one practice test, stumble when the writing style or the AI model changes. Paid products lean different ways: some miss AI text, some wrongly accuse people. Apps that promise to “humanize” AI writing so detectors cannot see it often fail against the strongest tools - unless they change what the text actually says.
A red flag means look closer. It does not mean proven guilty. Hold that line. The rest of this piece is why.
Plain words for the main ideas
AI text detector - a program that guesses whether a person wrote a passage or an AI wrote it.
AI / chat model (LLM) - tools like ChatGPT, Claude, or Gemini. They guess the next word. They are not a search engine that “knows” facts.
Wrong accusation (false positive) - the detector says “AI wrote this,” but a person wrote it. In school, this is the scary mistake.
Missed AI (false negative) - the detector says “a person wrote this,” but AI wrote it. Cheating can slip through.
Practice-test score (in-domain) - the tool is tested on writing that looks like what it already saw while learning.
New-world score (out-of-domain) - the tool is tested on different kinds of writing than it learned from (news vs essays, short vs long, and so on).
New-AI score (cross-generator) - the tool is tested on text from AI models it never saw while learning.
One big accuracy number (F1) - a single grade that mixes “when it says AI, how often is it right?” and “of all the AI texts, how many did it catch?” Good for ranking tools on one test. Bad as proof for your essay.
Humanizer - a tool that rewrites AI text so a detector (or a reader) is more likely to say “human.” Ads often say “undetectable.” Strong detectors often still notice.
How predictable the words are (perplexity) - if the next word is easy to guess, perplexity is low. Smooth AI writing often looks “too easy to guess.”
How uneven that predictability is (burstiness) - people often mix short blunt sentences with long ones. AI writing can sound flatter. Weak detectors watch for that flatness.
The core problem: a great score on the wrong test
In 2026, researchers (Pudasaini and coauthors, arXiv 2603.23146) built detectors from simple writing stats anyone can explain: how many paragraphs, how varied the words are, how much a file zipper can shrink the text. Repetitive, predictable writing often compresses smaller - AI drafts can look “zip-friendly” that way. They plugged those stats into ordinary machine-learning models - the same broad family of tools that power many apps, not magic.
On one public contest (PAN-CLEF 2025), their best setup scored about 97% (F1 0.9734). On another (COLING 2025), about 80% (F1 0.8025). For this style of tool, those are near the top of the public boards.
Then they switched the tests on purpose.
| Learned from | Tested on | Rough score |
|---|---|---|
| Contest A (PAN) | Contest A again | about 95–97 |
| Contest A | Contest B (COLING) | about 67–77 |
| Contest B | Contest B again | about 76–80 |
| Contest B | Contest A | about 78–83 |
One clear drop: a common off-the-shelf machine-learning model (XGBoost - a standard tool researchers plug data into) went from about 97 on Contest A to about 67 on Contest B. It did not suddenly get stupid. The writing changed - length, topics, how the AI samples were made. The cheap tricks it learned stopped working.
They also used a method called SHAP (it ranks which writing stats mattered most to the model). On Contest A, it cared a lot about how mixed the grammar patterns were. On Contest B, it cared about paragraph count and how zip-able the text was. If the tool had learned a real “AI voice,” it should care about the same clues everywhere. It did not. It often learned quirks of that contest’s data - not a universal “AI wrote this” signal.
The 97% was real. It was also a practice-test grade, not a courtroom stamp.
Two kinds of mistakes
A high overall score can hide which mistake the tool makes. For schools, that difference is the whole story.
Even on Contest A (the easy matching test), one of their models still wrongly labeled about 6% of human text as AI (and missed about 2% of AI text).
Turnitin has said its document-level wrong-accusation rate is under 1% (for documents where it marks about 20% or more as AI-written). One percent of millions of student papers is still a large pile of people flagged for nothing.
Short text is worse. In one deep look at errors, many mistakes piled up on short passages. Texts with basically one paragraph often got pushed toward “AI.” AI text with many paragraphs looked more “human.” Layout and length pretended to be proof of authorship. They are not.
Smoke alarms that hate short rooms are still smoke alarms. They are not proof of arson.
New AI tools break old detectors
Worth a short rewind: Contest A and Contest B were frozen in time. Chat tools were not.
The same researchers tested text from AI models released after the training data was locked: DeepSeek V3.2, GPT-5.2, and Gemini 3 Pro. Every sample was AI, so the only question was how often the detector missed it.
| New AI writer | Miss rate - model trained on Contest A | Miss rate - model trained on Contest B |
|---|---|---|
| GPT-5.2 | 23.25% | 3.42% |
| DeepSeek V3.2 | 0.76% | 3.51% |
| Gemini 3 Pro | 0.43% | 18.40% |
Same detector family. Three different AI writers. Totally different blind spots. Yesterday’s 97% does not travel free to tomorrow’s chat model.
Paid tools on one shared test
Pangram Labs sells a detector and publishes a technical report (arXiv 2402.14873). On their own test of 1,976 documents across 10 writing types and 8 AI models, they report about 99% accuracy and claim far fewer errors than DetectGPT and big commercial tools on that set. Treat the headline as the vendor’s own report until someone else repeats the test.
More useful: how rivals fail on the same papers.
| Tool | What it tends to do | Number from that shared test |
|---|---|---|
| GPTZero | Prefer “human” | Misses about 10% of AI text |
| Originality | Prefer “AI” | Wrongly flags about 9% of human text |
| Pangram (their own report) | Claims both error types stay low | about 99% accuracy |
Teachers usually care more about not accusing the wrong student than about catching every AI paragraph. GPTZero leans that way - and still leaves a lot of AI unmarked. Originality leans the other way - so more real human work needs a person to clear it. Neither pattern is “broken” in the abstract. Both are choices about which mistake hurts less.
People who write English as a second language get hit hard by some tools. One famous study (Liang and coauthors, arXiv 2304.02819) showed commercial detectors flagging TOEFL practice essays as AI - average wrong-accusation rates above 60% on that 91-essay set. Pangram says it got 0% wrong accusations on the same set. That is the company’s own claim on a known hard set - check it, do not treat it as settled law.
Three ways detectors work (and why “make it sound human” often fails)
Many people think detectors only hunt for AI clichés. That is only part of it - and it is the part marketers sell hardest.
Simple “how surprising is this word?” tools - If the words are too easy to guess, and the rhythm is flat, the tool says AI. A rewrite that makes the text bumpier can fool these weak tools.
Style-stat tools - They count things like paragraph breaks and how zip-able the text is. Breaking one long block into more paragraphs can fool those cues on one contest. The research above shows those cues break when the writing world changes.
Heavy trained tools (like Pangram) - They learn from huge piles of real human writing vs AI writing. The question is less “does this feel human?” and more “was this produced the way chat AIs produce text?” You can change the voice and still leave that fingerprint.
| Kind of detector | What it watches | What a surface rewrite often does |
|---|---|---|
| Word-surprise tools | Predictable, flat writing | Often lowers the AI score |
| Style-stat tools | Paragraphs, zip size, grammar mix | Can trick one contest; fails on new worlds |
| Heavy trained tools | AI “fingerprint” of how the text was made | Usually barely moves |
How a strong tool like Pangram learns
Why dig into training here? Because it explains why “make it sound human” often fails against this class of tool.
- Collect lots of writing that is known to be human (mostly from before 2021, so AI did not sneak in).
- For each human piece, ask an AI to write a twin - same topic, similar length, sometimes the same first line. Call that a mirror. Now the tool cannot cheat by learning “this topic means AI.”
- After a while, most twins are too easy. So they hunt for hard cases: human texts the current tool wrongly calls AI. They make new AI twins of those, and train again. Repeat.
On a huge fresh test set the model had never seen during training, they say wrong accusations fell from about 2.29% to about 0.02% after that loop. The lesson they train for is: same story, different author process (person vs AI). A rewrite that is still written by an AI is still the AI process.
What happens when you try to “humanize” against a strong detector
A live test against Pangram’s API on 2026-07-23 checked that claim. Score near 1.0 means “almost certainly AI.” Score near 0 means “almost certainly human.” Real human sample text scored about 0.003. AI sample text sat near 0.994.
| What we tried | Pangram AI score | Did the meaning stay the same? |
|---|---|---|
| Original AI text | 0.994 | - |
| One Claude rewrite | 0.95 | Partly (~68% meaning kept by automated check) |
| Best of six style rewrites | 0.992 | Mostly (~72% meaning kept) |
| Translate to French, then back to English | 0.994 | Almost yes (~99% meaning kept) |
| Four rounds of guided rewriting | 0.993 | No - meaning collapsed (~43% kept) |
| Replace about 70% of sentences with old public-domain human sentences (essay) | 0.035 | Meaning diluted (~64% kept) |
| Same mix idea on a news sample | 0.101 | Meaning diluted (~73% kept) |
Read the table slowly. The rewrite that kept meaning best (translate and translate back) did not fool Pangram at all. The loop that made the writing feel most “human” wrecked the meaning and still scored 0.993. The only approach that got under 0.5 (called human) swapped in large amounts of unrelated human sentences. That changes the content. It is not a clean “same essay, now undetectable.”
Researchers who study these attacks talk about a three-way trap: hide from the detector, keep the meaning, and keep the cost low. You almost never win all three. One paper that frames it this way is TH-Bench (arXiv 2503.08708). The live numbers above are one corner of that trap.
Limits of that live test: samples were short; some sample texts labeled “human” still scored like AI, so those samples were poor baselines; nobody tried attacks that look inside Pangram’s private model. Treat the table as one careful black-box story, not a law of nature.
Before you trust a detector
Treat it like any automated grading aid (same spirit as Five metrics for an agent eval pipeline):
- Ask what the score was measured on. Writing like yours? Language like yours? AI tools your people actually use?
- Ask for both mistake rates. “99% accurate” hides whether the tool wrongly accuses humans or quietly misses AI.
- Try it on your own sample. Real student essays, support emails, blog posts - skip relying on Wikipedia-shaped text alone.
- Re-check when AI models change. Yesterday’s detector can go blind on tomorrow’s chat tool.
- “I ran a humanizer” is not a clean pass. Against a strong tool, a rewrite that keeps the meaning often still looks like AI. A sudden drop in AI score may mean the meaning was watered down.
- For high stakes, keep a person in the loop. A flag is a reason to ask for drafts, notes, or a short talk - not an automatic verdict.
If you also use AI to score other AI writing, the same caution applies to LLM-as-a-judge: a number without a clear rubric and a look at failures will mislead you politely.
What to do with a red flag
Back to the opening: smoke alarm, not court.
A 97% contest score can mean “did well on this pile of test papers,” not “proved AI authorship.” A high AI score after a rewrite can still mean “an AI still wrote this,” even when the voice feels personal.
Before you punish anyone for a flag, run a small batch of known human writing from your world through the tool and count wrong accusations. For a wider view of scoring without worshipping one number, read Five metrics for an agent eval pipeline.
Sources
| # | Claim cluster | Source |
|---|---|---|
| 1 | In-domain F1 ~97 / cross-domain drop / SHAP / cross-generator miss rates | Pudasaini et al. - Why AI-Generated Text Detection Fails (arXiv 2603.23146) |
| 2 | Pangram bench (1,976 docs); GPTZero / Originality error bias; TOEFL 0% claim; hard-negative training | Pangram Labs - Technical Report on the Pangram AI-Generated Text Classifier (arXiv 2402.14873) |
| 3 | Commercial detectors biased against non-native English / TOEFL essays | Liang et al. - GPT detectors are biased against non-native English writers (arXiv 2304.02819) |
| 4 | Turnitin document-level false-positive rate under 1% (with 20%+ AI threshold) | Turnitin - Understanding AI writing detection: False positive rates |
| 5 | Humanizer trade-off: evade vs meaning vs cost | TH-Bench - Evaluating Evading Attacks via Humanizing AI Text (arXiv 2503.08708) |
| 6 | Live Pangram API humanizer checks (2026-07-23) | Author-run black-box harness against Pangram; not an independent third-party study |