AsianScientist (Jul. 20, 2015) – Scientific journals should insist on more robust experimental processes, say biologists after reviewing nearly 900,000 experiments. Their findings have been published in PLOS Biology.
The team found that non-blind experiments—that is, where scientists knew which samples they were recording—averaged a 27 percent stronger result than blind trials. However, their review suggests that less than one in four experiments used blind data recording.
“We found that non-blind papers tended to exaggerate differences between the experimental group and the control group,” said lead researcher Dr. Luke Holman, from the Research School of Biology at The Australian National University (ANU).
“For example, a non-blind trial of a new drug might conclude that it is way more effective than a placebo, when in fact the drug’s true effect is rather modest, simply because the researchers’ expectations biased the results.”
In the largest study of its kind, the team analysed nearly 900,000 papers from the PubMed life sciences database, using automated ‘data mining.’ They also—in a blind trial, of course—compared 83 pairs of evolutionary biology papers on similar topics, in which the data was collected blind in one, and not in the other.
The team also found that non-blind studies rejected the null hypothesis more strongly, said Holman. “Non-blind studies more confidently concluded that differences between treatment and control groups were real, and not just due to chance variation.”
Co-researcher Dr. Megan Head, also from ANU Research School of Biology said self-reflection is important. Scientists are aware of their biases, and use techniques such as blind trials to minimize them, but the pressure to get things done faster leads to some people skimping on experimental design, she said.
“Science is still the best method we have for understanding the world, and we have to keep working to make it better,” she added.
Holman and his colleagues believe that journals should insist on blind trials more strongly, perhaps by making prominent statements to authors and peer reviewers about the necessity of using blind trials.
“Many researchers don’t realize that their expectations can introduce such strong bias, and so they don’t feel the need to work blind,” he said.
The article can be found at: Holman et al. (2015) Evidence of Experimental Bias in the Life Sciences: Why We Need Blind Data Recording.
———
Source: Australian National University.
Disclaimer: This article does not necessarily reflect the views of AsianScientist or its staff.
