Questionable research practices are ways of conducting scientific research that distort data and findings. They came to wide attention in the early 2010s, when a study investigating false-positives in psychology showed that researchers were more likely to find evidence that an effect exists than to correctly find evidence that it does not. Ultimately, the study concluded that this bias could push false-positive rates above 60%.
Two of the most common examples of questionable research practices are p-hacking and HARKing, and these practices are contributing to the widespread replication crisis in scientific publishing. Authors may sometimes manipulate data until they provide a positive or significant outcome or rewrite a study’s hypothesis based on the results collected. Together, p-hacking and HARKing inflate and exaggerate findings, meaning that subsequent studies aiming to replicate the findings often fail to do so.
However, despite the serious effects of these questionable research practices, some instances are not deliberate fraud, like fabrication and falsification, and this distinction is important to observe. In this article, we will explain what p-hacking and HARKing are exactly, why they may occur without any intent to mislead, why the problems they cause matter, and how you can avoid them in future through preregistration.
What is the p-value?
In statistical research, the p-value (probability value) is a number between 0 and 1 that measures the extent to which your data contradicts the assumption that there is no statistical significance between variables in an experiment (a null hypothesis).
What the p-value in your study ultimately demonstrates is the likelihood of obtaining your results purely by chance, so long as your null hypothesis is correct.
The p < 0.05 threshold
To measure what is statistically significant data, there needs to be a threshold that researchers can compare their results against.
In statistical research, this threshold is p < 0.05. This means that there is a 5% or lower probability of the data occurring by chance if the null hypothesis is true.
What is p-hacking?
P-hacking, also known as data dredging, is a questionable research practice involving the artificial creation of statistically significant results. The researcher cherry-picks data to support or provide a positive outcome, thus manipulating the overall picture that all the data present.
Common examples of p-hacking include:
- Stopping data collection once a result is identified as statistically significant.
- Dropping outliers of changing test conditions.
- Testing multiple outcomes or covariates and reporting only those that “work” with the hypothesis.
The “garden of forking paths” problem
The “garden of forking paths” problem in statistics similarly inflates false-positives in research, but it differs because the researcher is not intentionally manipulating the data by testing different variations.
Instead, what this problem refers to is the fact that each analytical choice made in the study, such as what metric to focus on, which parts to examine, and what to do with missing data, etc., would have an impact on your final conclusions. And if you make these decisions after seeing the data, this inflates the false-positive rate.
This is why the issue of replication is so problematic in scientific publishing. Two scientists may come to different conclusions following the same experiment because different analytical choices have been made in the attempt to find statistically significant results.
What is HARKing?
HARKing (or Hypothesizing After the Results are Known) is the practice of presenting a post hoc hypothesis as if it was an a priori hypothesis. Or, in other words, presenting a hypothesis based on your results so that it positively conforms to the findings.
Why it is a problem
HARKing as a questionable research practice contributes to the replication crisis by distorting the scientific literature, making data look like the result of a preconceived hypothesis.
But HARKing is symptomatic of a wider issue in scientific publishing: publication bias and the pressure on researchers not just to publish, but to publish studies demonstrating statistical significance.
How to avoid it
One of the easiest ways to avoid HARKing is to undergo preregistration. This is the formal process of specifying your research plan in sufficient detail in advance of performing the study.
Preregistration was taken up widely in psychology and the social sciences in the early 2010s in response to the replication crisis. Now, preregistration serves as part of the scientific record, explicitly detailing the hypothesis of the study and the researcher’s intentions going into the experiment.
There are also other ways to avoid HARKing, such as:
- Clearly separating confirmatory and explanatory analyses.
- Reporting every analysis run.
- Sharing data and code, i.e., by writing a data availability statement.
Preprints and transparency
Another practical way to add transparency to your research process after analysis is by posting your scholarly manuscript as a preprint.
Posting on preprint platforms such as Preprints.org makes your manuscript visible to the research community early, and the commenting feature allows other researchers to question analytic choices in the open. Versioning provides a comprehensive public record of how a manuscript has changed, and authors can link their preregistration, data, and code directly in the preprint.
Because Preprints.org welcomes negative and non-significant results, researchers have a route to share their work without needing a “publishable” p-value, which removes one of the pressures behind p-hacking.