What Is This?
Most warnings about generative AI focus on one failure mode:
AI -> fabricates a fact -> you believe a false thing
That is real, but it is the shallow version of the problem. A sharper idea comes from a 2026 preprint by Kamitani and Shirakawa, which names a subtler failure: phantom evidence.
Phantom evidence is not a false claim. It is a false sense of how hard the claim was to produce.
weak result + fluent presentation -> feels like strong evidence
When a person sees a polished, high-resolution, well-argued output, they unconsciously infer that it must have survived a large search — many alternatives were considered, many ways it could have failed did not. That inference is often wrong. The output may reflect a narrow generator, hidden trial-and-error, data leakage, or a system grading its own work.
Phantom evidence is the gap between how much scrutiny an output feels like it survived and how much it actually did.
Why Does It Matter?
For anyone using AI to search, synthesise, and validate — which is increasingly how research gets done — the danger is not just being told something false. It is mistaking fluency for confirmation.
A convincing paragraph is a presentation fact, not an evidence fact. The question that matters is not "does this sound rigorous?" but:
what search space was actually sampled to produce this,
and how many ways could it have failed but didn't?
If the answer is "a narrow one" or "we can't tell," the polish is decoration, not support.
The Core Model: Persuasiveness Is Not Evidence
Evidential weight comes from surviving opportunities to fail. A result is strong when it could easily have come out differently and didn't — when the space of alternatives was large and the constraint was hard.
An impressive-looking output tells you almost nothing about that space on its own. A model can produce a beautifully reasoned defence of a claim and an equally beautiful defence of its opposite. Fluency is cheap and roughly constant; it does not scale with truth.
evidential weight ~ (size of search space) x (hardness of the constraint)
persuasiveness ~ fluency, resolution, confidence of presentation
These two axes are nearly independent. AI has driven the cost of the second toward zero while leaving the first untouched.
The Old Problem: False Positives Before AI
None of this is new. Science already had a false-positive problem long before language models.
Ioannidis argued in 2005 that most published research findings are false, because under low prior probability, small samples, bias, and selective reporting, a "significant" result is more likely to be noise dressed as signal. Simmons, Nelson, and Simonsohn showed that ordinary "researcher degrees of freedom" — flexible stopping rules, dropping outliers, adding covariates — can manufacture statistically significant nonsense. Gelman and Loken sharpened this with the "garden of forking paths": you do not need to cheat deliberately. Implicit analytical branching, each step feeling reasonable from the inside, is enough to produce a false positive. Smaldino and McElreath added the incentive layer: methods that reliably produce publishable results can spread through a field even when they make it less truth-tracking.
The through-line: impressive results have always been cheaper to produce than true ones.
What AI Changes
Generative AI does not invent these failure modes. It industrialises them. It lowers the cost of producing:
- polisded explanations for almost any claim,
- plausible-looking citations,
- synthetic "replications" and variations,
- and self-graded outputs that score themselves well.
before: producing a convincing false positive was expensive
after: producing a convincing false positive is nearly free
The result is that the world fills with high-fluency, high-confidence artefacts whose evidential backing has not increased at all. The signal-to-persuasion ratio drops.
The Evidence Ceiling
A key discipline follows: more fluency, resolution, or detail does not raise evidential strength.
A longer answer is not a stronger answer. A more confident tone is not more support. A citation you did not check is not evidence — it is a claim about evidence. Past a point, added polish is added presentation, and presentation has a ceiling above which it tells you nothing new about truth.
Where This Shows Up
The concrete risk is treating fluent synthesis as independent confirmation. Watch for it when:
- a brief cites "multiple sources" that all trace back to one origin,
- an agent reports high confidence in its own output,
- a summary reads as authoritative because it is well-written, not because it was checked,
- a model both generates and evaluates the same claim.
"Multiple sources" that are not actually independent are one source wearing several coats.
How To Use This
A working checklist:
- Separate three things every time: the claim, the evidence, and the explanation. Fluency lives in the explanation; weight lives in the evidence.
- Check source independence, not source count.
- Ask what search space was actually sampled.
- Discount self-graded outputs — a system scoring its own work is not evidence.
- Ask for the negative case: what would this look like if it were false?
- Treat polish as presentation, not validation.
What This Does Not Prove
The term "phantom evidence" comes from a single 2026 arXiv preprint, so treat the label as proposed, not canonical. This does not claim AI uniquely corrupts science; the stronger, better-supported claim is that AI reduces the cost of persuasive false positives. The underlying mechanisms — false positives, analytical flexibility, p-hacking, publication incentives, and NLG hallucination — are independently and well established. And "hallucinated citation" is only one narrow example; the broader failure is mistaken evidential weight.
Recall Questions
- What is the difference between fluency, search space, and evidential weight?
- Why is a more detailed AI answer not automatically a stronger one?
- What makes "multiple sources" fail to count as independent support?
- What is the single question to ask of any convincing output?
Sources
- Kamitani & Shirakawa, "Phantom Evidence: How and Why Generative AI Manufactures False Positives in Science," arXiv:2607.25991 (2026) — proposes the phantom-evidence concept; a preprint, treat as proposed.
- Ioannidis, "Why Most Published Research Findings Are False," PLoS Medicine (2005), DOI: 10.1371/journal.pmed.0020124.
- Simmons, Nelson & Simonsohn, "False-Positive Psychology," Psychological Science (2011), DOI: 10.1177/0956797611417632.
- Gelman & Loken, "The Garden of Forking Paths" (2013), Columbia University working paper.
- Smaldino & McElreath, "The Natural Selection of Bad Science," Royal Society Open Science (2016), DOI: 10.1098/rsos.160384.
- Ji et al., "Survey of Hallucination in Natural Language Generation," ACM Computing Surveys (2023), DOI: 10.1145/3571730.