Image (above): Distinguished Professor Susan Scott, Editor-in-Chief of Classical and Quantum Gravity. Credit: ANU.

Journal editors’ inboxes fill with AI-generated papers. Fake datasets underpin published research. This is not a vision of the future, it’s the increasingly complicated world of scientific publishing. 

How much AI is too much? 

Journals generally allow authors to use AI to improve grammar, spelling and readability. This doesn’t usually require disclosure, but more substantive use of AI must be documented. 

Ultimately, AI cannot be an author. Nature, for example, states that AI can only be used “as a supporting technology; scholarly judgement, accountability, and responsibility always remain human”.

Despite these rules, growing numbers of AI-generated papers are making their way into journal submissions. And they’re getting harder to spot. 

“AI writing has certainly improved dramatically in recent years,” says the Australian National University’s Distinguished Professor Susan Scott, Editor-in-Chief of Classical and Quantum Gravity. 

Professor Adrian Barnett of QUT, who studies research funding, peer review and research integrity, agrees. He and his colleague experimented with AI to “write a paper in half an hour with a couple of prompts that would be good enough to get published”.

For editors, this means extra work. They are now using AI to screen papers for possible AI-generated content. “For instance, we check whether references include recent work, because large language models often omit the latest studies,” says Scott. 

Publishers must also keep pace with the threats posed by generative AI (genAI), such as paper mills. “These commercial enterprises fabricate research and research papers at increasing scale,” says Alice Black, head of Editorial and Research Integrity at CSIRO Publishing.  

AI is also changing how research is assessed. In a recent joint statement, the Australian Research Council and the National Health and Medical Research Council say that genAI may support the grant application process, but that responsibility, expert judgement and decision-making must remain with humans.

Barnett agrees that AI peer review should “only ever be an adjunct” to human review, not a replacement. He actually thinks AI could remove some of humanity’s “huge status bias issue” when it comes to selecting grant applicants. “AI should be more interested in the idea, not where it comes from,” he notes. 

When pressure meets convenience

Regardless of rules, humans are drawn to convenience – while researchers are under constant pressure to produce. What happens when AI makes this easier?

“I worry that most academics are just going to produce more rather than better, because we’re in this ridiculous competitive environment of publish or perish,” says Barnett.  

He also describes how trust in science is being compromised by fake but realistic-looking datasets. For instance, he found that three datasets on the online repository Kaggle, including one “comically bad dataset” used to train clinical models, had been used in 135 published papers. 

How is this possible? “Unfortunately, a lot of journals don’t ask questions,” Barnett explains. “And a lot of researchers who are just looking for a dataset to get a paper don’t ask questions, either.”

Publishers have much to gain from the efficiency of AI tools, but they may need to be more effective at filtering out low-quality papers. Small publishers have an advantage here. “As matters of research misconduct grow in complexity … we can have human eyes on every paper and provide manual assessment for any that raise concerns,” says CSIRO’s Black.

Why human judgement matters 

Ultimately, trust in published science depends on human integrity. AI can help us summarise and express ideas, but it must never decide outcomes. 

As Scott puts it: “If humans are going to retain our role as the pre-eminent entity on this planet, then we will need to preserve our function of carrying out the heavy-duty brain-work, including scientific creativity and ingenuity.”

Working out how we preserve that function might be the biggest question of all. 

– Written by Kath Kovac