For centuries, science has rested on a deceptively simple promise: a claim is not true because an authority says it is true; it is true because the evidence can be inspected, tested, challenged, and reproduced. That promise is now under pressure from a new force inside the research ecosystem itself—artificial intelligence.
AI is no longer sitting outside the laboratory as a productivity tool. It is writing literature reviews, generating code, summarizing papers, drafting grant proposals, assisting in peer review, designing molecules, predicting protein structures, and producing figures. In its best form, it is a scientific accelerator. In its worst form, it is a confidence machine: fluent, fast, persuasive—and sometimes wrong.
The result is a new trust crisis. The question is no longer only whether a paper is peer-reviewed. The harder question is whether the evidence is real, whether the citations exist, whether the images were generated or manipulated, whether the data can be traced, and whether the work can be reproduced by humans outside the original team.
In the AI age, scientific trust cannot depend only on reputation. It must depend on verification.
The alarm is not theoretical. A 2026 arXiv study titled LLM hallucinations in the wild audited 111 million references across 2.5 million papers in arXiv, bioRxiv, SSRN, and PubMed Central. The authors estimated that 146,932 hallucinated citations appeared in 2025 alone, after the widespread adoption of large language models. The study also warned that existing preprint and journal safeguards caught only a fraction of these errors.
That finding is striking because citations are supposed to be the nervous system of science. They connect claims to prior evidence. When a citation is fake, the paper does not merely contain a typo; it creates a false trail of authority. A reader may assume the claim has a foundation. A reviewer may miss the error. A future AI model may ingest the paper and repeat the falsehood with even greater confidence.
A separate analysis published as correspondence in The Lancet found that fabricated citations in biomedical literature rose sharply from 2023 to 2025. Reporting on that analysis, Retraction Watch noted that the audit covered nearly 2.5 million papers and found a twelvefold increase in fabricated citations over two years. CIDRAP’s summary of the same review reported that 4,046 fabricated references appeared in 2,810 papers, with the rate rising from four per 10,000 papers in 2023 to 51.3 per 10,000 by the end of 2025.
This is how scientific pollution spreads: not always through dramatic fraud, but through small, plausible, machine-generated errors that enter the record at scale.
The New Threat Is Not AI Itself, but Unverified AI
AI has already shown extraordinary scientific value. The 2024 Nobel Prize in Chemistry recognized David Baker for computational protein design and Demis Hassabis and John Jumper for developing an AI model that helped solve the long-standing challenge of predicting complex protein structures. That achievement showed what responsible computational science can do when models, validation, domain expertise, and reproducible evidence work together.
The danger, therefore, is not that AI is entering research. The danger is that AI is entering research faster than the trust infrastructure around research can adapt.
AI can expand the frontier of science. But without verification, it can also expand the frontier of error.
Publishers are now trying to define boundaries. Elsevier’s generative AI policy allows authors to use AI-assisted technologies in manuscript preparation, but only with oversight and disclosure. It also warns that editors should not upload confidential manuscripts into generative AI tools and should not use such systems to make editorial decisions, because the final judgment must remain human and accountable. Wiley similarly states that authors must disclose the use of AI technologies when submitting material to its journals.
The emerging consensus is clear: AI cannot be an author because it cannot take responsibility. Human researchers remain accountable for every claim, citation, figure, method, and conclusion. Research-integrity guidance from the German Research Ombudsman captures this principle directly: AI does not qualify for authorship, and AI use must be declared transparently.
But disclosure alone is not enough. A paper can disclose AI use and still contain fabricated citations. A figure can be labeled as AI-assisted and still misrepresent the underlying data. A methods section can mention a model and still omit the prompt, version, seed, data source, preprocessing steps, or validation protocol needed to reproduce the result.
The next phase of scientific publishing will therefore move from AI disclosure to AI auditability.
Reproducibility Becomes the Central Test
The reproducibility crisis existed long before ChatGPT. Many fields have struggled with small sample sizes, selective reporting, inaccessible data, fragile statistical methods, and incentives that reward publication volume over verification. AI intensifies those weaknesses.
With generative tools, a weak manuscript can be made to look polished. A fabricated reference can look scholarly. A low-quality review article can sound authoritative. A synthetic figure can appear publication-ready. A paper mill can generate variations of the same fraudulent structure faster than editors can detect them.
Springer Nature and other publishers have joined industry-wide integrity initiatives to screen for compromised manuscripts. The STM Integrity Hub provides a cloud-based environment that allows publishers to check submitted articles for research-integrity issues, including integrations with third-party tools. In 2025, Springer Nature donated an AI-powered tool designed to detect AI-generated nonsense text in manuscripts to the STM Integrity Hub, reflecting the publishing industry’s attempt to fight machine-assisted paper mills with machine-assisted screening.
This is a necessary response, but it also reveals the scale of the problem. Scientific publishing is becoming an arms race: AI generates, AI screens, humans adjudicate, and the literature sits in the middle.
The future of research integrity will not be decided by whether journals ban AI. It will be decided by whether journals can verify evidence faster than falsehood can be produced.
A reproducible paper in the AI age should not merely say, “We used AI.” It should answer harder questions: Which model? Which version? Which prompts? Which data? Which parameters? Which human checks? Which code? Which independent validation? Which failed attempts were excluded? Could another lab regenerate the same result?
Without those answers, AI-assisted science risks becoming beautiful but brittle.
Peer Review Was Not Built for Machine-Scale Error
Peer review remains essential, but it was designed for a slower world. Reviewers are busy scientists, often unpaid, asked to evaluate novelty, methodology, interpretation, statistics, and relevance. They were not historically expected to verify every citation title across multiple databases, inspect every image for synthetic artifacts, audit every dataset lineage, and reproduce every computational result.
The NeurIPS citation controversy illustrates the challenge. A 2026 arXiv paper analyzing 100 fabricated citations in NeurIPS 2025 papers reported that those citations appeared in 53 accepted papers, despite expert review. The study argued that fabricated references can exploit multiple verification shortcuts at once, making them hard to detect through traditional review.
The lesson is uncomfortable but important: peer review is not a forensic system. It is a quality-control process, and quality control now needs stronger tooling.
That may mean automated citation verification before submission. It may mean mandatory data and code availability statements. It may mean provenance records for images. It may mean model cards for AI-assisted analysis. It may mean stronger penalties for unverifiable work. It may mean journals treating reproducibility not as an optional ideal but as a condition of publication.
What Must Change
Scientific trust in the AI age will require a new operating model built on five principles.
First, human accountability must be explicit. AI can assist, but named human authors must remain responsible for every claim. “The model generated it” cannot become a defense.
Second, citations must be machine-verified before publication. If large-scale AI tools can fabricate references, journals and repositories need automated checks against CrossRef, PubMed, Semantic Scholar, OpenAlex, and publisher databases before manuscripts move forward.
Third, figures and images need provenance. AI-generated or AI-altered figures should be declared, and scientific images should preserve source files, processing history, and audit trails wherever possible.
Fourth, data and code must become inspectable by default. Reproducibility cannot survive if datasets are unavailable, code is missing, prompts are undisclosed, or model versions are unspecified.
Fifth, research incentives must shift from quantity to credibility. The publish-or-perish culture helped create the market for paper mills. AI will make that market cheaper and faster unless institutions reward rigorous, reusable, transparent work over raw publication counts.
The Trust Layer of Science Is Becoming a Product Requirement
For universities, funders, journals, and research platforms, trust is no longer an abstract value. It is becoming infrastructure.
A modern research system may need dashboards that show whether citations are verified, whether datasets are available, whether code runs, whether images have provenance, whether AI use is disclosed, whether peer-review history is transparent, and whether independent replications exist. In other words, the scientific paper may evolve from a static PDF into a living evidence package.
That shift will be difficult. It will increase costs. It will slow some publications. It will frustrate researchers who already face administrative overload. But the alternative is worse: a scientific record where real discovery and synthetic authority become indistinguishable.
The AI age does not make science obsolete. It makes scientific discipline more important. The core values remain unchanged: evidence, skepticism, transparency, replication, and accountability. What changes is the level of proof required to defend those values.
Science has always asked, “Can this be proven?” The AI age adds three urgent questions: Was it humanly accountable? Was it real? Can it be reproduced?
The institutions that answer those questions clearly will define the next era of credible research. Those that do not may find themselves overwhelmed by a literature that looks scientific, sounds scientific, and cites science—but cannot be trusted as science.
