Evaluating Reproducibility in Psychology: A Machine Learning-Based Large-Scale Study on Predicting Replication Success

I. Introduction

A. Background

Reproducibility in scientific research, particularly in fields like psychology, is often compared with other domains where optimization models have been successfully employed to address complex issues. For instance, in the field of search advertising, Zhou et al. explored how reinforcement learning can optimize ad rankings and bidding mechanisms, providing a framework that combines machine learning with economic models to enhance system performance in real-world scenarios [1]. Similarly, applying machine learning models in psychology to assess the likelihood of replication success opens up scalable opportunities to address the reproducibility crisis across a vast body of research, just as Zhou’s approach in advertising led to more accurate ad placements and cost-efficiency. This analogy highlights the potential for machine learning to predict outcomes in psychology, improving reliability and scientific rigor on a large scale.

B. Objectives

The primary objectives of this paper are:

• To analyze reproducibility across different subfields of psychology.
• To evaluate the influence of research design, sample size, author experience, and citation impact on replication success.
• To provide a scalable, machine learning-based method for estimating replication likelihood, and to discuss its implications for scientific rigor.

II. Related Work

A. Traditional Replication Studies

The concept of reproducibility in science traditionally relies on manual replications, wherein researchers attempt to replicate the experimental conditions and findings of a published study. However, such studies are resource-intensive and limited in scope [4]. A notable example is the Reproducibility Project: Psychology (RPP), which attempted to replicate 100 studies and found that only 39

B. Machine Learning Models for Reproducibility Prediction

Predictive modeling techniques have emerged as a scalable solution for assessing reproducibility. These models use features such as study design, sample size, p-values, and textual data from the research paper to estimate the likelihood of replication success. One prominent example is the use of random forest and logistic regression models to predict replication outcomes based on narrative descriptions of the research design and results [6].

Machine learning models have proven effective in addressing complex prediction problems across various industries. For instance, Zhou et al. (2024) applied reinforcement learning to optimize search advertising, combining Generalized Second-Price auctions with dynamic bidding strategies [1]. Their model successfully improved cost-efficiency and user satisfaction by adapting to real-time feedback, balancing short-term profit maximization with long-term user engagement.

The success of Zhou et al.’s model is relevant to predicting research reproducibility, where similar trade-offs must be managed between factors like study design, sample size, and statistical rigor. Zhou et al.’s work illustrates how machine learning can optimize outcomes in dynamic environments with multiple inputs, providing useful insights into building models that estimate replication success in psychology.

Furthermore, the scalability and adaptability of Zhou et al.’s approach offer promising solutions to large-scale reproducibility studies. Much like in advertising, where automation enabled efficient decision-making, machine learning models can assess replication likelihood across thousands of studies, addressing the inherent limitations of manual replication efforts.

III. Methodology

The integration of machine learning models to assess reproducibility draws inspiration from similar approaches in other fields. For example, Zhou et al. (2024) implemented a reinforcement learning-based model in the context of search advertising, which successfully optimized ad placements by simulating real-world interactions in an offline environment and fine-tuning the model through online feedback [1]. In our study, we adopt a similar methodology by applying machine learning models, such as random forests and logistic regression, to estimate replication success in psychology. Like the reward function in Zhou’s model, which balances click-through rate and platform revenue, our model optimizes predictive accuracy by leveraging study features such as sample size, p-values, and textual data. By training on replicated studies, we ensure that the model learns to adjust based on empirical evidence, much like reinforcement learning optimizes based on user feedback.

A. Dataset and Sample Characteristics

Our dataset includes 14,126 papers published between 2000 and 2020 in six leading psychology journals: Psychological Science, Journal of Experimental Psychology, Journal of Personality and Social Psychology, Developmental Psychology, Clinical Psychology Review, and Organizational Behavior and Human Decision Processes. The dataset covers a wide array of subfields, including clinical, cognitive, developmental, organizational, personality, and social psychology [8].

The data was gathered from publicly available sources and included key features such as the number of authors, author affiliations, citation counts, media attention, and funding sources. Additionally, the papers were categorized based on research design (experimental vs. non-experimental) and methodological approach.

B. Machine Learning Model

To predict the likelihood of replication success, we employed an ensemble of random forest classifiers and logistic regression models. The models were trained on a dataset of 388 manually replicated studies, which served as the ground truth for replication success or failure [9]. The predictive models used both numerical and textual data, with the textual data converted into feature vectors using Word2Vec. The following steps were involved in the model training process:

• Word2Vec Embedding: Each word from the abstracts of the papers was converted into a 200-dimensional vector based on semantic similarity. This technique captures the contextual meaning of words, allowing for the quantification of narrative elements in the paper [10].
• Feature Selection: Key features such as study design, sample size, p-values, and textual vectors were used as inputs to the machine learning models [11].
• Training and Validation: The models were validated using 10-fold cross-validation and achieved an average area under the curve (AUC) score of 0.74, comparable to results from predictive markets [12].

C. Performance Metrics

We evaluated the performance of our models using the following metrics:

• Precision: The proportion of correctly predicted replication successes to total predicted successes.
• Recall: The proportion of actual replication successes that were correctly predicted.
• AUC: A summary measure of the model’s accuracy across all possible classification thresholds.
• F1 Score: A harmonic mean of precision and recall, providing a balanced measure of the model’s performance [13].

IV. Results

A. Replication Rates Across Subfields

Our analysis revealed significant differences in replication rates across subfields. Personality psychology exhibited the highest estimated replication rate (77%), followed by organizational psychology (68%). In contrast, social psychology and developmental psychology showed the lowest estimated replication rates, with averages of 38% and 36%, respectively [14]. These findings highlight the variability in research rigor and replicability across different areas of psychology.

B. Impact of Research Design

Consistent with previous research, we found that non-experimental studies had significantly higher replication scores compared to experimental studies. Non-experimental studies had an average replication score of 0.50, while experimental studies scored an average of 0.39 [16]. This difference can be attributed to the inherent challenges in replicating controlled experimental conditions in real-world settings.

C. Author Expertise and Citation Impact

We observed a positive correlation between replication success and author expertise, measured by the cumulative number of publications and citations [17]. Papers authored by researchers with higher publication counts and citation impact were more likely to replicate successfully [18]. However, we found no significant relationship between university prestige and replication success, suggesting that institutional reputation is not a reliable proxy for research quality.

D. Media Attention and Replication Failure

Interestingly, our results indicated that papers receiving higher levels of media attention were less likely to replicate successfully [11]. This finding aligns with previous research suggesting that sensational or counterintuitive findings, which are more likely to attract media coverage, are also more prone to replication failure [10].

V. Discussion

A. Implications for Research Practices

The findings from this study have important implications for improving research practices in psychology. First, the significant variability in replication rates across subfields suggests that targeted interventions, such as replication incentives and methodological training, should be tailored to specific research areas. For instance, subfields with lower replication rates, such as social and developmental psychology, may benefit from stricter peer-review processes or the use of pre-registration protocols to enhance research rigor.

Second, the positive correlation between author expertise and replication success highlights the importance of mentorship and collaborative networks in promoting high-quality research. Encouraging senior researchers to collaborate with early-career scientists could improve overall research quality and reproducibility.

Finally, the negative association between media attention and replication success suggests that public dissemination of research findings should be approached with caution. Journalists and media outlets should be encouraged to report on research with a clear understanding of the limitations and uncertainties inherent in the findings.

B. Limitations and Future Research

While this study provides valuable insights into the reproducibility of psychology research, it is not without limitations. First, our models were trained on a limited number of manually replicated studies, which may introduce bias into the predictions [13]. Second, the use of textual data for prediction, while scalable, may not capture all relevant factors influencing reproducibility [15].

Future research should aim to expand the scope of replication studies to include a broader range of subfields and research designs. Additionally, integrating machine learning models with other predictive tools, such as replication markets, could provide more robust estimates of reproducibility.

VI. Conclusion

This study represents the first large-scale, machine learning-based assessment of reproducibility in psychology. By analyzing over 14,000 papers across six subfields, we provide a comprehensive overview of replication rates and their associated factors. Our findings highlight the importance of methodological rigor, author expertise, and the need for caution when disseminating research findings to the public. As psychology and other social sciences continue to grapple with the replication crisis, machine learning offers a scalable solution for assessing and improving scientific rigor.