Scientific progress is driven by discovery. But the problem with discovery is that it naturally privileges positive results. Researchers might try to be recognized for research that achieves success and contributes to knowledge-building in their field and beyond, and publishers may potentially put a bit more focus on work that advances their fields. Unfortunately, this situation has led to what is known in scientific publishing as publication bias.
In this article, we’ll explain what publication bias in research is, why it matters, the efforts employed to detect and resolve it, and what you can do as a researcher when dealing with negative outcomes.
What is publication bias?
Publication bias is the tendency of researchers, journals, and publishers more likely to prioritise statistically significant or positive results at the expense of negative, null, or inconclusive results.
There are certain systemic reasons for this bias. For example, it has been demonstrated that publishers showcase “editorial bias”, which consists of favouring positive results and publishing positive-results studies before negative-results studies in journal schedules. Funders’ commercial interests can also influence which research findings become publicly available.
This systemic positive-results bias leads to a secondary phenomenon: selective outcome reporting. This is where researchers choose to report only specific measures and results based on their confirmatory or statistical significance.
The file drawer problem
Selective outcome reporting has resulted in what psychologist Robert Rosenthal in 1979 termed the “file drawer problem”. He claimed that journals were typically filled with the 5% of studies that demonstrate Type 1 errors (or positive, statistically significant results), and the remaining 95% of studies that show nonsignificant results stayed hidden or unpublished in physical or digital “file drawers”.
The evidence
Since Rosenthal’s claim, there has been supporting evidence of the file drawer problem in scientific publishing.
In 2014, a study published by Annie Franco, Neil Malhorta, and Gabor Simonovits investigating 250 peer-reviewed proposals for social sciences experiments found that only 10 out of 48 null results were published. In contrast, 56 out of 91 studies demonstrating positive results made it into journals.
Another study goes further, claiming that positive outcomes have risen so much in recent decades that negative results are disappearing entirely from most disciplines and countries.
The evidence is clear. But what justifies the concern?
Why publication bias and the file drawer problem matter
The progress of knowledge depends on a comprehensive picture of an issue. For example, the refinement of a certain type of medicine depends not only on understanding the positive impacts already present in the drug or dosage, but also on what the medicine does not have any effect on at all.
To not report negative, null, or even inconclusive results has subsequent problems. It runs the risk of distorting the scientific literature and inflating effect sizes across disciplines. This leads back to the wider issue of reporting transparency, which in turn has helped foster the reproducibility crisis in scientific publishing, or the difficulty independent researchers face in replicating the findings of other studies. If only significant measures and outcomes are reported, the bigger picture remains unclear, and the road to replication incomplete.
Subsequent issues of keeping negative results hidden include misinformed clinical and policy decisions and wasted research effort when researchers repeat studies that have already failed but have not been published.
Solutions to the problem
There are certain solutions to publication bias in research and the file drawer problem. These include:
- Preregistration.
- Registered Reports.
- Openly sharing negative or null results.
What unites these solutions is their emphasis on transparency.
Preregistration
Preregistration is the process of confirming your research plan, hypothesis, and methodology in advance of collecting and reporting your data and findings. This reduces publication bias in research by preventing questionable research practices such as p-hacking and HARKing, which involve the artificial creation of statistically significant results and presenting a post hoc hypothesis as if it was an a priori hypothesis so that it positively conforms to your data and findings, respectively.
Registered Reports
Registered Reports shift peer review to the formulative stage of the research process. They are a form of report that assesses the importance and quality of the research question and methodology prior to experimental analysis.
If approved, the journal then commits to publishing the paper regardless of the outcomes reported.
Open sharing of negative or null results
Transparently sharing negative or null results means that no research is left hidden in a physical or digital drawer. Because of this, accurate meta-analyses and systematic reviews can be carried out on all the available literature. This reduces artificial inflation of positive results, leading to a more comprehensive picture of a specific issue.
Finally, researchers can rest assured knowing that their work does not go to waste and can be published whether it demonstrates statistically significant outcomes or not.
What individual researchers can do
A null result left in a file drawer helps no one. Fortunately, there are avenues you can take as a researcher wanting to report and share negative or non-significant results. One of the quickest and most direct avenues is posting a preprint.
Preprint servers such as Preprints.org explicitly welcome negative or non-significant results.
In the above figure is a study posted on Preprtints.org. The researchers indicated no conclusive evidence to support a bidirectional causal genetic liability between periodontitis and Parkinson’s Disease. However, the results were deemed scientifically important nonetheless, and the research manuscript was eventually published as a peer-reviewed article.
Every research finding has value. For more information about transparently sharing your results with a global audience via open access, see our Instructions for Authors.