Scalability and Accuracy Assessment of Frequent Pattern Mining Algorithms Applied to Large-Scale Hospital Databases

1. Introduction

The rapid digital transformation in healthcare has led to massive growth in clinical data, particularly with the widespread adoption of electronic health records (EHRs). Extracting meaningful knowledge from these datasets is critical for improving diagnosis accuracy, identifying disease correlations, and supporting preventive care strategies. Frequent Pattern Mining (FPM) algorithms, such as Apriori, FP-Growth, and ECLAT, are among the most commonly used methods for discovering co-occurrence patterns within large medical datasets. However, as hospital databases scale to millions of records, traditional FPM techniques face challenges in computational efficiency, memory requirements, and accuracy. This study aims to systematically evaluate and compare the scalability and accuracy of three leading FPM algorithms applied to large-scale hospital databases.

2. Background and Related Work

Frequent pattern mining has long been used for market basket analysis but has recently gained significant attention in healthcare analytics. Apriori is known for its simplicity but suffers from high computational cost. FP-Growth improves performance by using compact tree structures, reducing the need for repeated scanning. ECLAT, using vertical data formats, performs well for certain dataset types but struggles with dense datasets. Previous research highlights the role of FPM in predicting disease progression, understanding comorbidities, and improving clinical decision support systems (CDSS). However, limited studies assess how these algorithms perform on truly large and complex hospital datasets. This gap motivates the present research.

3. Methodology

3.1. Dataset Description

Experiments were conducted using:

• A synthetic hospital dataset of 5 million patient records.
• A real-world EHR dataset obtained from an open-source medical repository, including diagnoses, lab tests, and medication histories.

3.2. Algorithms Evaluated

• Apriori: Candidate generation-based algorithm.
• FP-Growth: Tree-based method using frequent pattern trees.
• ∙ ECLAT: Vertical format mining through itemset intersection.

3.3. Evaluation Metrics

• Scalability: Runtime performance as dataset size increases.
• Memory Efficiency: Peak memory usage during execution.
• Accuracy: Ability to identify clinically valid patterns (measured with support and confidence).
• Processing Overhead: Number of database scans and intermediate structures.

4. Results

4.1. Scalability

• FP-Growth scaled efficiently to millions of transactions with minimal overhead.
• Apriori showed exponential growth in runtime, becoming impractical for datasets larger than 500,000 records.
• ECLAT performed moderately well but experienced slowdowns with dense medical datasets.

4.2. Memory Consumption

• Apriori consumed the most memory due to extensive candidate generation.
• FP-Growth had the most balanced memory usage.
• ECLAT consumed minimal memory in sparse datasets but struggled in dense ones.

4.3. Accuracy

All three algorithms identified valid clinical associations; however:

• FP-Growth produced the most patterns with high confidence and lift.
• ECLAT performed best in detecting patterns in sparse datasets (e.g., rare diseases).
• Apriori produced fewer but highly precise associations.

4.4. Pattern Quality

FP-Growth discovered the widest diversity of disease co-occurrence patterns, making it suitable for rich hospital datasets.

5. Discussion

The evaluation demonstrates that FP-Growth is the most robust, scalable, and accurate algorithm for large-scale hospital data mining. Apriori, though useful for smaller datasets, is not practical for modern healthcare databases. ECLAT has niche advantages in sparse data environments but lacks consistency in dense clinical data settings. Healthcare institutions seeking to implement frequent pattern mining for clinical decision support should consider dataset characteristics, computational resources, and pattern complexity when choosing an algorithm.

6. Conclusion

This study provides a detailed assessment of the scalability and accuracy of three major frequent pattern mining algorithms applied to hospital databases. FP-Growth emerges as the most suitable algorithm for large and complex EHR datasets due to its high scalability, reduced memory consumption, and superior accuracy. Future work may involve hybrid models, parallelization strategies, and integration with AI-based clinical prediction systems.