DeskDealFinder: Application for Online Shopping Optimization on Desktop Products with Web Scraping

1. Introduction

In Peru, e-commerce has taken a prominent place in the economy. Consumers can purchase products from the comfort of their homes and compare prices with just a few clicks. This boom has increased the number of virtual stores, leading to a price dispersion that makes it difficult for consumers to make decisions. In this context, the need arises for a tool that facilitates price comparison in an effective and efficient way.

DeskDealFinder is a platform designed to optimize online shopping for desktop products. Using web scraping techniques, it collects price data from various Peruvian online stores and presents this information in a clear and accessible way for users. By centralizing and comparing prices, the platform not only simplifies purchasing decisions, but also promotes transparency and competitiveness in the market.

Web scrapping automates the manual process of human visits to lists of web sites periodically to search and store data. Copying the data manually and pasting it into files is a tedious and time-consuming job. Automated web scrapping tool does the same job in a fraction of the time. Web scrapping software can be configured to work with any website or can be custom built for a specific website. However, generic web scrapping software may not offer any option to extract the required content due to its fixed template.[1]

The rapid growth of the Internet has led to an exponential increase in the digitization, making it difficult for users to find relevant information efficiently. Information retrieval systems are information retrieval systems are instrumental in addressing this challenge by organizing and retrieving relevant data from vast and diverse web resources. As as a technique for automatically extracting data from web pages, web scraping has become indispensable in the development of robust and efficient information retrieval systems. and efficient information retrieval systems. [2]

The main element of this is a Web Scraper (Web Scraping) that uses the company’s website URL as a starting value to completely traverse the company’s website domain and extract text from the pages and associated hyperlinks. The company websites are crawled using specially developed crawlers written in Python that recursively traverse the entire website. recursively traverse the entire website and collect text and link data from each web page. The crawlers are configured to collect only text data and ignore images, PDF files and other multimedia files. Since a considerable portion of website content may be stored on dynamically generated content pages when visiting a new page on the company’s website. page on the company’s website. [3]

Web page content stores data in a dedicated database. As a result, from the disaggregated dataset where a unit of analysis is a single URL that represents the company as a whole, we build a dataset that is as a whole, we build an overall company-level dataset. This dataset contains all the raw data and therefore raw data and, therefore, the indicators that are most relevant for the analysis. The structured schema ensures interoperability between data sources [3].

Webscraping, as described in this article, cannot be applied to applications. applications, it only applies to Internet pages. [4]

Companies and digital businesses rely on recommender systems to increase their profits, and we, as users, rely on them because they reduce information overload and make our lives easier.[5]

First, each of the data sources is accessed in turn and new links or records are identified, before downloading or deleting all available data from the identified records and links. This process is scheduled to occur daily so that new records can be captured as quickly as possible after they are posted. This reduces the risk of web links being modified or deleted before they are captured. [6]

It is important to note that this data set can be used for can be used for tasks such as classification and natural language processing. natural language processing. [7]

We use standard web scraping techniques and limit our requests in order to minimize any impact on the platform. Our goal is to collect data responsibly. [8]

the fundamental role of web scraping to drive innovation, enable more effective management of human capital dynamics and improve results. [9]

The sheer volume of online data makes manual collection and processing impractical for researchers. manual processing impractical for researchers. With the proliferation of online databases, the Internet becomes a critical resource that accentuates a critical resource that accentuates the need for fast and accurate data retrieval. The solution is called web scraping. Web scraping is the scraping is the process of implementing algorithms. [9]

2. Justification

With the growing consumption of online shopping, consumers face a challenge when trying to compare prices of similar products in different stores, which can lead to poor purchasing decisions. DeskDealFinder addresses this issue by collecting and comparing prices, offering an accessible solution for users.

In addition, the platform contributes to market transparency, allowing retailers to adjust competitively and benefiting consumers with better deals. This transparency can also incentivize stores to maintain competitive prices and improve service quality.

This project implements the web scraping technique in the Peruvian context, addressing specific challenges such as variability in web page design and legal restrictions. The integration of technologies such as HTML, CSS, PHP and Python in the development of DeskDealFinder demonstrates a practical and scalable solution that can be adapted to other contexts and markets.

Web crawling is an essential part of how the web works. It is through this process verifies whether the stores in question can be scraped as such. Therefore, companies generally do not allow web crawling. The policy of websites that cannot be scraped is usually described in a specific resource on the website called “robots.txt” file. During the web scraping process, we look for “robots.txt” files, which indicate which sections of the website are accessible to crawlers. website are accessible to crawlers. Therefore, we obey the policy described there. [3]

Acquire new data sources and methods to provide a systematic path to a fast, reliable and accurate representation of company activities, while maintaining structure and harmony. [10]

3. Methodology

The type of research for this study is applied, since it seeks to develop a practical tool to improve the efficiency of online shopping through the use of web scraping techniques. Given that it is an applied research that should account for the level of acceptance or rejection developed by the user to the technology. [11]

The approach of this research is quantitative and descriptive. Metrics such as the number of lines of code will be used to evaluate the complexity of development and the efficiency of web scraping.

For the population and sample, research was done in virtual stores specialized in tableware.

Probably 15 virtual stores, mostly Peruvian, of Peruvian origin. The analysis of these stores will be conducted to evaluate the effectiveness of the deskdealfinder tool and price distribution.

The main objective is to evaluate the effectiveness of web scraping to capture and compare prices, and the influence of deskdealfinder on users’ purchasing decisions.

The study will use a sample of 3 online stores as a guide, to ensure diversity in size and popularity.

The requirements were examined, taking into account the needs of consumers who purchase tableware online through online stores located in Peru. Speed and accuracy were determined by both functional requirements, such as the ability to extract and compare prices, and non-functional requirements.

For the instruments and techniques, Peruvian stores focused on desktop products will be the main source of data collection, with priority given to Python-based web scraping software.

Structured observation will be used. Web scraping will be used to extract products and prices and a questionnaire will be used to assess user satisfaction and perception using deskdealfinder.

Data collection will be pilot tested and a consistency check between different data sources will be performed to ensure accuracy and reliability. In addition, the effectiveness of the code will be evaluated using metrics.

The procedure for the development and evaluation of DeskDealFinder included several key steps, detailed below:

A detailed requirements analysis was carried out, considering the needs of consumers who buy desktop products online in Peruvian virtual stores. Functional requirements, such as the ability to extract and compare prices, and non-functional requirements, such as system speed and accuracy, were identified.

The MoSCoW method was used to prioritize the requirements:

• Must have: Ability to extract price data from multiple stores, friendly user interface, database to store the data.
• Should have: Filters to compare products, support for multiple product categories.
• Could have: Integration with payment platforms, personalized recommendations.

SCRUM was adopted as the agile methodology to manage the project. The main activities included:

• Sprints: 4 7-week sprints were planned to develop increments of software functionality.
• Sprint review: At the end of each sprint, deliverables were reviewed with stakeholders and feedback was obtained.
• Retrospectives: Evaluations at the end of each sprint to identify areas for improvement.

To evaluate the efficiency of code development, the following metrics were used:

• Average Processing Time: The amount of code written was measured to evaluate productivity.
• Success Rate per Page: They were counted to ensure good code documentation.
• Response Rate per Store: Monitored to evaluate code structure and readability.

The following key functionalities were developed and evaluated:

• Data Extraction: Use of web scraping techniques to collect pricing data of desktop products from several Peruvian online stores.
• User Interface: Development of a user friendly interface using HTML, CSS and PHP to improve the user experience.
• Database: Implementation of PhpMyAdmin to store and manage the collected data.

3.1. Análisis de datos

Data collected through web scraping will be analyzed using spreadsheets. The data will be categorized based on the order and content of relevant variables, such as product price, store and date of collection.

Once the data is obtained, a descriptive analysis will be performed to provide an overview of price dispersion and market trends.

A descriptive table of the variables used in the study is presented below:

Table 1. Variables used in the analysis

Variable	Description
Success Rate	Peruvian web sites analyzed
Success Rate per Page	Percentage of success of scraping per Page
Average Processing Time	How long the algorithm took to scrape per page
Number of Successful Attempts per Site	How many scraped products were successfully obtained

Table 1. Variables used in the analysis

Variable	Description
Success Rate	Peruvian web sites analyzed
Success Rate per Page	Percentage of success of scraping per Page
Average Processing Time	How long the algorithm took to scrape per page
Number of Successful Attempts per Site	How many scraped products were successfully obtained

The data analysis will conclude by identifying price trends and savings opportunities for consumers. This will be based on price dispersion metrics and the efficiency of web scraping in gathering accurate and up-to-date data.

3.2. Ethical considerations

Since the data collected came from online stores and not from individuals, no additional ethical considerations were required. But we did take into account their privacy policy and whether they had protection which was not skipped in this research.

4. Results

5. Discussions

We wonder about the integration of gamification into current and future digital trends. [12] As the amount of information available on the web increases, so does the task of locating and analyzing it, so performing this task manually can be costly in terms of the time and effort invested. Although search engines and database engines can help to find the required information, in large digital infrastructures where search results are in the thousands - or more - new tools are needed to obtain the searched content effectively. This paper proposes the application of Web Scraping. [13]

The main web scraping tools available in the market and compares their features and functionalities. A specific tool is selected to demonstrate its use in obtaining data. The use of digital techniques such as web scraping for massive data de-loading. Thanks to this technique, unstructured information is converted.[14]

Web pages are good tools for communication and for sharing information with others, but also for participation. The method is developed in two stages (i) web scraping, which allows to obtain the links of information from web pages and to download their data, and (ii) data analysis with data cleaning. [15] In 2018, a group of researchers from the Technological University of Panama, created an algorithm using the R language to extract data from GS using the Web Scraping technique. from GS using the Web Scraping technique. [16]The algorithm, which integrates several functions, uses this unstructured data mining technique to scan the code HTML code of a web page and dynamically extracts the data the data that is displayed structured or unstructured and transforms it into table format for into table format for debugging, displaying, accessing and exporting. and subsequent export. [17] There is data that was captured by web scraping and processed using algorithms and techniques for the using algorithms and techniques for the analysis of massive network data sets. [18].

Its objective is to serve as a navigable index for the quick location of information of interest to readers, as well as a visual map that highlights the characteristics of the knowledge network that these data form through the interconnection of relationships between its nodes. This device has been created by extracting data from the magazine’s website using web scraping programs. [19] In a web scraping performed from January 1 to March 31, 2015, 1 222 320 news items were captured which demonstrates the massive amount of data that can be obtained. [20] Defined queries are executed sequentially using the method of web-scraping method, until all search possibilities are exhausted. [21] First of all, it must have starting links for each platform, from which it makes a recursive search of all the courses it finds, and stores them in a list so that they can be subsequently worked by the web scraper. later worked by the web scraping. [22] This document illustrates the use of data science techniques for the creation of a database with information on prices and sales characteristics. [16]