Efficient Big Data Processing and Recommendation System Development with Apache Spark

II. Methodology

A. Dataset

The dataset used in this project was sourced from the GroupLens website, a research initiative from the Department of Computer Science and Engineering at the University of Minnesota. GroupLens provides a variety of datasets, including MovieLens, HetRec2011, WikiLens, Book-Crossing, Jester, and EachMovie. For this research, the MovieLens 20M dataset was selected because it is recommended for new research applications.

The MovieLens 20M dataset contains six CSV files and one text file, summarizing 27,278 movies and 138,493 randomly selected users. In terms of the dataset structure, the first file, genome-scores.csv, consists of three columns: movieId, tagId, and relevance. The relevance column represents how strongly a specific tag matches a particular movie, with values ranging from 0 to 1. The second file, genome-tags.csv, contains two columns: tagId and tag. This file provides user-generated labels for movies, which are often descriptive words or phrases. The third file, links.csv, contains movieId, imdbId, and tmdbId, which allow cross-referencing of movies with external databases such as IMDb and TMDb.

The fourth file, movies.csv, contains three columns: movieId, title, and genres. The title includes the movie name along with its release year in parentheses, and the genres column lists the categories associated with each movie in a pipe-separated format. The fifth file, ratings.csv, has four columns: userId, movieId, rating, and timestamp. The rating ranges from 0.5 to 5.0, and each user has rated at least 20 movies. Finally, the sixth file, tags.csv, records user-generated tags along with timestamps. A text file named README provides a detailed description of the dataset and its structure.

B. Data Preprocessing

Data preprocessing was a crucial step in this research to ensure the integrity and usability of the dataset [12]. One significant issue encountered was with movie titles containing commas, which disrupted the parsing of the movies.csv file. For example, the title “Godfather, The (1972)” caused the genre column to be misread as part of the title. To address this, the preprocessing included validating data formats and correcting misaligned fields. Specifically, the input file was filtered to only include rows where all columns satisfied specific data type requirements. For instance, the project ensured that the movie ID consisted of numeric characters and that ratings adhered to a valid floating-point format.

A test case was designed to identify such issues by filtering records for specific movie IDs, such as movieId 858, and checking whether the genre column displayed the correct value. In the case of movieId 858, the expected genre, “Crime|Drama,” was initially replaced by part of the title. By addressing this misalignment, the preprocessing step ensured that subsequent analyses could be conducted without errors. Other than this issue, the dataset was found to be clean and ready for further processing.

C. Spark Algorithm

The analysis relied on Apache Spark for its distributed computing capabilities [13], which facilitated efficient processing of the large dataset. First, the ratings.csv, movies.csv, and genome-scores.csv files were read into Spark as RDDs. Each file was parsed by splitting rows into arrays based on commas. A case class was defined for each dataset to provide structure, allowing elements to be assigned appropriate names and data types such as strings or doubles. The RDDs were then converted into DataFrames for ease of manipulation, and their schemas were printed to verify the data structure.

Second, headers were removed from each dataset by applying filters to exclude rows that did not conform to expected patterns. This step ensured that all subsequent computations were based on valid data. For instance, the ratings dataset was filtered to retain only rows where the movieId and rating columns matched numeric patterns.

Third, key transformations were applied to derive meaningful insights. The ratings dataset was used to calculate statistics such as the count, minimum, maximum, mean, and standard deviation of ratings. Additionally, genres and tags were joined with ratings data to provide a comprehensive view of each movie's characteristics. For example, the project performed an intersection of movies with five-star ratings and movies with high tag relevance to identify the top-ranked films.

Finally, the top 10 movies were extracted by ranking movies based on their ratings and relevance. These results were saved to an HDFS output directory for further verification. To optimize performance and scalability, Spark's configuration settings were adjusted, including increasing the number of executors and memory allocation. This ensured that the system could handle larger datasets with minimal degradation in performance.

D. Performance Evaluation

The methodology also included an evaluation of the algorithm's scalability. The dataset, which was approximately 150 MB in size, required four minutes to process under the initial configuration. To accommodate larger datasets, the project proposed increasing the number of executors to 30 and allocating 1 GB of memory per executor. This adjustment significantly improved processing speed, demonstrating the robustness of the Spark framework for handling big data tasks.

III. Experimental Results

A. Statistics of the Dataset

After applying data preprocessing techniques, the dataset was analyzed to extract key characteristics, uncover trends, and gain insights into its structure. This section highlights the distribution of genres, user rating patterns, and tag relevance, offering a comprehensive view of the dataset.

Tag correlation analysis provides crucial insights into user preferences and the latent attributes of movies. The most relevant tags, including "original," "mentor," and "great ending," indicate strong user engagement and specific expectations for movies. By leveraging these insights, recommendation algorithms can refine movie suggestions by prioritizing films with strong correlations to high-rated tags. Incorporating tag correlation into collaborative filtering models enhances recommendation accuracy by considering the qualitative aspects of user feedback rather than just numerical ratings. As illustrated in Figure 1, the distribution of genres across movies is highly diverse, but certain genres dominate the dataset. Drama is the most prevalent genre, reflecting its universal appeal and storytelling depth. This is followed by Comedy, which caters to a wide audience seeking lighthearted entertainment, and Thriller, known for its suspenseful and engaging narratives. The prominence of these genres suggests they are central to the dataset and may align with popular user preferences. Other genres such as Action, Romance, and Horror also hold notable shares, underscoring the variety in movie offerings and user interests.

The user rating patterns, visualized in Figure 2, show a clear positive bias. Most ratings are concentrated in the range of 3.5 to 4.5, with a peak around 4.0, indicating that users tend to rate movies favorably. This skew toward higher ratings could be attributed to the dataset's nature, where movies that receive poor ratings may be less frequently watched or rated. The relatively small proportion of ratings below 2.0 suggests that either the movies in the dataset are generally well-regarded or that users are reluctant to provide harsh ratings. The histogram in Figure 2 provides a detailed representation of this trend, helping to understand the overall sentiment in user feedback.

An analysis of tag relevance, presented in Figure 3, reveals the most dominant tags associated with movies. These tags provide an additional layer of context to the dataset, highlighting specific qualities and attributes that resonate with users. The tags “original,” “mentor,” and “great ending” emerged as the most relevant, indicating that users often value originality, the presence of a guiding or inspiring character, and a satisfying resolution. Other notable tags include “emotional,” “unexpected twist,” and “visually stunning,” which reflect diverse user preferences and experiences. This analysis underscores the depth of user engagement and the role of specific attributes in shaping movie perceptions.

In summary, the dataset demonstrates a rich diversity in genres, a tendency toward positive ratings, and meaningful engagement with specific movie attributes through tagging. These insights lay the foundation for further analyses, such as recommendation systems or predictive modeling, by emphasizing the factors that most significantly influence user preferences.

B. Results and Performance from Spark Algorithm

The primary goal of this project was to identify the top ten movies in the state of Minnesota based on their movie IDs, titles, and genres. However, due to certain data inconsistencies, some movie titles and genres in the output required clarification. For instance, the movie with MovieId 94833 was incorrectly listed with the title "The (2012)" and genre "The (2012)." The corrected title is “Pirates! Band of Misfits, The (2012),” and the corrected genre is “Adventure|Animation|Children|Comedy.” Similarly, for MovieId 858, the title should be “Godfather, The (1972)” instead of “Godfather,” and its genre should be “Crime|Drama.” Another notable example is MovieId 78772, which was listed as “Twilight Saga: Eclipse,” with an incomplete genre. The corrected title and genre are “Twilight Saga: Eclipse, The (2010)” and “Fantasy|Romance|Thriller|IMAX,” respectively. Each movie in the list typically has at least two genres, with some having as many as six. Table 1 provides the corrected list of the top eight movies in Minnesota, including their Movie IDs, titles, and genres.

To ensure the correctness of the output, test cases were implemented and evaluated. For example, the first movie in the top 10 list, “Pirates! Band of Misfits, The (2012)” with MovieId 94833, was verified using the following test case: “val check2 = step3.filter(x => x._ 1.toInt == 94833).take(10)”. The test returned: “check2: Array[(String, String)] = Array(94833, “Pirates! Band of Misfits, genre: The(2012)”)”. This output confirmed the presence of the movie in the top 10 list, validating the algorithm’s accuracy.

Apache Spark played a pivotal role in efficiently handling the large-scale MovieLens dataset. By leveraging distributed computing, Spark significantly reduced data processing time compared to traditional approaches. The use of DataFrames enabled optimized memory utilization, while Spark’s SQL- based transformations improved data querying efficiency. Additionally, Spark’s built-in machine learning libraries facilitated the implementation of collaborative filtering techniques, ensuring an optimized recommendation process. The dataset used in this project had a size of 150 MB, and the entire process executed within 4 minutes. To handle larger datasets efficiently, performance scaling was evaluated. When doubling the dataset size, execution time can be optimized by increasing the number of executors to 30 and allocating 1 GB of memory per executor. This scaling approach ensures that the Spark algorithm remains efficient even as the dataset size grows, highlighting its suitability for large-scale data processing tasks.

IV. Conclusion

This study presents a scalable and efficient approach to analyzing the MovieLens 20M dataset using Apache Spark, with a focus on identifying the top-ranked movies in Minnesota. The analysis of genres, user ratings, and tag correlations provides key insights into regional user preferences. Furthermore, integrating tag correlation analysis enhances the accuracy of recommendation systems by considering qualitative user preferences beyond numerical ratings. Apache Spark's role in this study demonstrates its effectiveness in large-scale data processing, significantly improving computational efficiency and scalability. The findings emphasize the importance of distributed computing frameworks in modern recommendation systems and highlight potential areas for further enhancement. Despite its strengths, this methodology has certain limitations. The reliance on collaborative filtering approaches may struggle to capture evolving user preferences or address the cold-start problem, where limited data for new users or items hinders recommendations. Additionally, the analysis does not incorporate external contextual factors, such as social media sentiment or real-time user interactions, which could further enhance predictive accuracy. Future research will explore advanced machine learning techniques, such as neural collaborative filtering, to improve personalization and adapt to non-linear user-item relationships. Moreover, integrating external data sources, including social media sentiment analysis and real-time user interactions, will provide a more comprehensive understanding of user preferences and trends. By addressing these aspects, future studies can further optimize recommendation systems and refine big data analytics techniques, ensuring more accurate and personalized content recommendations for users.