Information & truth. What’s the difference? I’ve always liked this analogy from the world of data science: data is information, but models are truth. Let’s start with the data. This image shows total monthly publications for a particular journal up until mid 2024: On its own, the data doesn’t tell us much that’s interesting. But a little bit of analysis can go a long way here.
The first rule of research integrity is always cite your sources. In fact, if there was only 1 rule, that would be it. Almost all research misconduct boils down to someone trying to take credit for work, discoveries, achievements, or ideas that aren’t their own.
Clear Skies’ data analysis shows that peer-review rejects papers where we flag concerns. When I buy something with a credit card, I tap my card on a card reader to make a payment. It’s very convenient. Every now and then, I might go into a shop where I’ve never been before. Then, when I tap my card, I am asked to enter my Personal Identification Number (PIN) to verify that I am me. Why does this happen?
Clear Skies is making genAI detection with Pangram available to subscribers in Oversight. We’re excited to see how this helps users to investigate research papers.
Clear Skies created the first index of research integrity: Oversight. We offer metrics describing research standards covering the entire research ecosystem. If you want to know more about metrics for your portfolio, get in touch. I sometimes get asked why I picked the name ‘Clear Skies’. I used to fly gliders. Gliders don’t have engines, so technically, they don’t actually “fly”, they just kind of fall out of the sky.
This post provides an update on some recent new features in Oversight and walks through an investigation. Given the pace of development of Oversight, I might update it in future as things change. The problem is time. It’s time spent on misconduct investigations. Such investigations are often complex and sometimes inconclusive — fraud is intentionally hidden after all.
Oversight, by Clear Skies, is the first complete index of journal integrity. Oversight takes complex data on research integrity and enables a straightforward and intuitive understanding of every journal in the world. We rank journals based on analysis of data from our extensively field-tested integrity tools.
TL;DR: The Papermill Alarm has grown to become a comprehensive mix of methods for detecting organised research fraud and simplifying a complex problem. But there’s something new. Something bigger and better from Clear Skies.
“Disruption” is what happens when a different technology develops to a point where it does what yours does well enough to meet the needs of your customers. That means that, when customers switch, you can’t compete without changing your product completely. There’s a brilliant essay by Michael Nielsen from 2009 which predicted that disruption in academic publishing was imminent. It’s still worth a read, even now.
It’s 1 January 2021. Dozing in bed, floating in that warm & fuzzy limbo between dreams and reality, and that’s where it hits him. That lightbulb moment. (At last!) He sits bolt-upright in bed, punches the air and shouts: “You know what, honey, I’m gonna start a papermill!”. She snorts awake; reluctantly conscious. Oh no. Not again. Not another hare-brained scheme… “…and, by the end of next year, we’ll publish 20,000 fake papers!”.
TL;DR: AI-generated-text detection ceased to be a good indicator of research fraud at the exact time that it became a good tool for generating fraudulent research. I remember finding ChatGPT usage in a research paper for the first time. It was April 2023, and a Twitter user pointed out that you could identify bots by their use of the phrase “as an AI language model…”. It’s a hallmark of ChatGPT usage.