Revolutionizing Engagement: The Fusion of Personalization and Augmented Reality

1. Introduction

Museums today aim to educate and entertain the public, not just preserve artifacts [1]. To enhance visitor experiences, many institutions are leveraging information technology, with personalization emerging as a key strategy in HCI research. We propose an AR fireworks system for creating DIY hybrid souvenirs at the Shanghai Science and Technology Museum. This system, comprising an app, QR code printer, and online gallery, enables children to learn scientific concepts through AR-simulated experiments. Adopting a STEAM education approach, our system visualizes abstract ideas and safely replicates dangerous processes. This paper outlines the design and reports on the real-world deployment of our system. The AR souvenirs demonstrate significant potential for customization, serving as personalized memory bridges between visitors and exhibitions.

2. Background

2.1. Personalization in the Museum Experience

Museums and galleries have embraced technology to enhance artifact presentation, significantly upgrading visitor experiences by improving education, entertainment, and research [1,2,3]. Successful technology adoption must address socialization and personalization concerns. Curators and designers now integrate interactive and personalized experiences, such as personal photo stories and interpretations [4]. Projects like “Seeing with New Eyes” and “Automics” show the effectiveness of hybrid souvenirs created by recording personal voices or images [5]. Despite their potential, many museum souvenirs remain monolithic and non-interactive. Innovative approaches, like “data souvenir recording”, aim to enhance memories and serve as time-space mediators [6]. Souvenir engagement varies, focusing the experience, aiding reflection, or capturing it as a memento.

2.2. Socialization in the Museum Experience

Museum visits are increasingly social, with souvenirs strengthening bonds and attracting visitors. To enhance souvenir excitement, museums can employ personalization, hybrid souvenirs, and multi-sensory experiences. Personalization results in unique souvenirs reflecting diverse backgrounds. Hybrid souvenirs, though engaging, show that physical gifts elicit more excitement than digital ones [5]. Physical objects accumulate personal histories and promote reciprocity. Multi-sensory experiences, such as AR gesture recognition, recorded messages, or scents, enhance emotional impact by evoking visit memories. These strategies create meaningful souvenir experiences, fostering connections between visitors, their social circles, and the institution.

2.3. Augmented Reality in Museum Education

AR and VR adoption in museums has increased, reflecting a push for digital innovation [7,8]. These technologies offer novel educational possibilities, simulating teaching environments and promoting interactive learning through virtual-real blending and three-dimensional immersion [9]. VR, exemplified by “The Met 360° Project”, offers immersive environments for remote visits but may detach from the physical museum space [8]. AR seamlessly merges virtual elements with the real world, offering higher personal and emotional engagement [10,11]. The “Creating Personalized Dinosaur Worlds” exhibition demonstrates AR’s potential, using AR glasses for gestural control of digital models. While AR glasses provide more immersion than mobile phones, they face accessibility and comfort challenges. Mobile phones, though less immersive, are more prevalent and cost-effective. AR glasses’ limitations, including single-user constraints and discomfort, hinder widespread adoption in learning environments.

3. Design

In this part, we describe the design of the AR fireworks system in detail including the application, QR code print system, and online gallery. We will describe the process, functionality, and topic selection. We focus on the application of AR and STEAM to educational functions, and the detailed design of DIY Souvenir for personalization and socialization.

3.1. AR Firework System

The proposed system serves children and parents visiting the science and technology museum, as well as souvenir vendors. We designed learning partners, including a small dragon, and engaging plots to attract children, the primary users. Children use smartphones for AR exploration of exhibits and interactive learning. Through a DIY fireworks activity, they gain knowledge. To keep their creations, children can purchase a physical fireworks launcher box from vendors. By attaching a unique QR code and uploading their model, children save their work as a souvenir. When reopened, the box displays the model with associated memories. Gesture controls allow rotation and fireworks activation. Users can also share their models in an online gallery.

Figure 1. Project Architecture.

Figure 1. Project Architecture.

Table 1. A storyboard of a visitor interacting in the exhibition and their use of the system. Users choose to purchase the hybrid souvenir and send it to others. They’ll learn knowledge in the process of DIY souvenirs.

Children and their parents came to the museum and entered the Information Age exhibition hall. They found that the exhibits were old and lacked interaction. 1. Enter the Science and Technology museum	They suddenly find a poster with a QR code, their parent scan it with their phone and then download the app called My Firework. 1. Scan the QR code 2. Download application	They opened the app, read the instructions, clicked the DIY button, and their learning partner appeared and played a plot animation. 1. Read instructions 2. Start DIY fireworks 3. Watch plot animation 4. Learning partner: little dragon
Users need to log in or register, and the system generates recommendations for routes to explore the exhibition based on their age. 1. Login/Register 2. Generate recommendations for routes based on age	Children follow the hints to find the materials for making fireworks. Then they find the Magnesiteore. 1. Explore the exhibits 2. Find material for fireworks	By scanning the pictures of the material around the ore, corresponding models and introductions will appear. The material is added to the backpack. 1. Scan the picture 2. Molecular models emerge 3. Add material to the backpack
Children and parents learn about fireworks and customize AR fireworks souvenirs together. 1. DIY fireworks 2. Learn knowledge 3. Parents buy the fireworks launcher box for the children.	They’ll print a QR code and attach it to the box. Then they can DIY even add audio and finally upload it. 1. Buy firework launch box 2. Attach QR code 3. DIY souvenir 4. Add audio Upload	They can keep this souvenir for themselves or give it to others. When the box is opened next time, the models with memories appear. 1. Give it to others/ Keep it 2. Open again 3. Show the AR model

Table 1. A storyboard of a visitor interacting in the exhibition and their use of the system. Users choose to purchase the hybrid souvenir and send it to others. They’ll learn knowledge in the process of DIY souvenirs.

Children and their parents came to the museum and entered the Information Age exhibition hall. They found that the exhibits were old and lacked interaction. 1. Enter the Science and Technology museum	They suddenly find a poster with a QR code, their parent scan it with their phone and then download the app called My Firework. 1. Scan the QR code 2. Download application	They opened the app, read the instructions, clicked the DIY button, and their learning partner appeared and played a plot animation. 1. Read instructions 2. Start DIY fireworks 3. Watch plot animation 4. Learning partner: little dragon
Users need to log in or register, and the system generates recommendations for routes to explore the exhibition based on their age. 1. Login/Register 2. Generate recommendations for routes based on age	Children follow the hints to find the materials for making fireworks. Then they find the Magnesiteore. 1. Explore the exhibits 2. Find material for fireworks	By scanning the pictures of the material around the ore, corresponding models and introductions will appear. The material is added to the backpack. 1. Scan the picture 2. Molecular models emerge 3. Add material to the backpack
Children and parents learn about fireworks and customize AR fireworks souvenirs together. 1. DIY fireworks 2. Learn knowledge 3. Parents buy the fireworks launcher box for the children.	They’ll print a QR code and attach it to the box. Then they can DIY even add audio and finally upload it. 1. Buy firework launch box 2. Attach QR code 3. DIY souvenir 4. Add audio Upload	They can keep this souvenir for themselves or give it to others. When the box is opened next time, the models with memories appear. 1. Give it to others/ Keep it 2. Open again 3. Show the AR model

Users will engage in an interactive learning experience while creating DIY fireworks. During the process, they can perform AR chemistry experiments as they collect materials. For example, when collecting magnesite, users can simulate the combustion of a magnesium strip. We offer two types of virtual experiments (Figure 2): a standard virtual scene and an AR experiment where props are integrated into the real environment. Dragging the magnesium strip near an alcohol lamp triggers it to burn brightly, illustrating the principle behind fireworks. Other experiments include metal flame color reactions and potassium hypochlorite decomposition.

After collecting all necessary materials, users can start crafting their fireworks by choosing from various types—spherical, line, triple color, and custom picture fireworks—as shown in Figure 2. Users can adjust and preview parameters (Figure 3), learning physics concepts along the way. The firework’s height is determined by its initial launch velocity, typically higher for larger fireworks for safety and aesthetics. The shape depends on the internal structure and pyrotechnic arrangement. Adjustable parameters like shape, speed, diameter, and color levels offer a realistic DIY experience.

Additionally, users learn about parabolas when creating linear fireworks. Traditional teaching methods struggle to capture motion trajectories, often relying on theory. In our simulation, the trajectory is captured in real-time, with a small dragon guiding users to adjust parameters like velocity and angle, allowing them to observe the dynamically adjusted parabola.

Thus, they have potential risks that can be solved by the AR technique. The parabolic experiment in middle school is so abstract that most students find it hard to understand. Our experimental simulation can offer help. Moreover, the variety of colors and shapes of fireworks provide opportunities for appreciation and personalization.

3.2. STEAM Education Approach

To enhance the learning experience in science and technology museums, we used the STEAM approach, integrating Science, Technology, Engineering, Art, and Mathematics to foster inquiry and critical thinking. In our application, “Science” is taught through the physical and chemical principles of fireworks, where users learn how launch velocity and explosion diameter affect the parabolic trajectory.

“Technology” is explored through AR, while “Engineering” is developed as users follow precise procedures to adjust firework parameters like speed, diameter, and angle. Creativity is encouraged through “Art” in DIY firework activities. This integrated approach offers a comprehensive and meaningful learning experience [12]. “Mathematics” is reinforced with charts and formulas that explain parabolas.

3.3. Augmented Reality and Natural Hand Gesture

Augmented reality, supported by the ARFoundation SDK, is utilized in various ways, including interacting with museum exhibits, presenting 3D molecular models, simulating experiments, guiding DIY firework creation, and detecting hand gestures[13]. We chose the phone as the display device for its convenience, despite its limitation of not being usable by multiple users simultaneously.

Object and image recognition are employed to locate materials needed for fireworks. We modeled these materials according to theoretical models, accounting for different ion and molecular structures, such as chemical bonds and electron pairs. Plane detection is applied in the DIY firework process, allowing users to scale models on their phones after plane detection. We enhanced user experience by enabling fireworks to be placed on various surfaces, such as cakes or buildings, rather than limiting placement to a single plane.

One of the most significant applications of AR in education is experimental simulation. Real chemical experiments pose safety risks, such as the dangers of open flames or exposure to toxic substances like barium chloride, which require proper protective equipment. By simulating laboratory environments and chemical experiments, AR helps avoid these risks while replicating the experimental phenomena of real-life situations.

Finally, natural hand gesture detection, supported by the ManoMotion SDK [14], is used to trigger the fireworks. ManoMotion supports five gestures: pick, pinch, click, grab, and release. We selected the release gesture to trigger the fireworks, as it mimics the user setting off fireworks from their hands, creating a natural and magical effect. The pinch gesture was chosen to control rotation, allowing the model to rotate clockwise when the camera detects a pinching motion and stop when the user releases it. This enables users to easily view souvenirs from multiple angles without needing to rotate the phone around the model. The pinch gesture was chosen for its convenience and precision, as it provides the most agile and natural control among the five gestures.

3.4. The Hybrid Souvenir with DIY Firework

This hybrid souvenir merges a virtual AR component with a physical element, offering Science Museum visitors a unique experience. The virtual aspect allows visitors to collect hidden materials throughout the museum to design a personalized firework. Customizable features include shape (based on uploaded images or text), color scheme, and launch background options like a castle, ocean, or grassland.

The physical component is an exquisite box available for purchase at the museum. Visitors can print the QR code of their fireworks design and attach it to this box. The process involves designing the fireworks using the AR application, purchasing the box, printing the QR code at the “Designer’s Cradle” exhibition hall, and attaching it to the box.

This souvenir experience is multi-sensory, encouraging visitors to engage beyond visual observation. To open the gift, visitors must perform a specific gesture in front of the camera; a correct gesture unlocks the gift. Additionally, visitors can record a voice message sharing their impressions of the museum, which will play alongside the fireworks show, enhancing their overall experience.

Furthermore, the souvenir serves as a keepsake of social connection. Visitors can send their personalized firework and a recorded voice message as a gift to others, potentially attracting them to the museum. They can also upload their fireworks to a public “firework gallery” in the application, where others can view, like, and comment on their creations, encouraging repeat visits and reliving memorable experiences.

3.5. System Architecture

Digital technology powers an interactive experience, seamlessly integrated into the system’s three-layer architecture. The Application Layer manages direct interactions, including the app, online gallery, and QR code printer. Users print a QR code, create a personalized gift, learn related knowledge, and optionally add their model to the digital gallery.

The Service Layer handles backend functions. User data is encrypted with the RSA algorithm, and a unique 16-digit QR code is generated and stored in the user’s repository. The content delivery system transfers digital components (audio, QR codes, models) between the back end and front end, while event processors manage tasks like customizing fireworks. The gallery management system oversees digital gallery data.

Finally, the Personalization Service visualizes and presents the model to the recipient by recognizing the QR code on the souvenir box.

We use AR technology to make the application more interesting, avoid the risk of dangerous experiments, and visualize abstract concepts. We designed a 3d cartoon image of a small dragon as a learning partner, added a plot, and built a task system based on it. The little dragon plays the role of guiding the learning steps, accompanying the player, and advancing the plot. Interacting with the little dragon in an AR environment adds to the fun of the application.

Figure 5. System Architecture.

Figure 5. System Architecture.

4. Findings

The My Firework app was deployed in the wild on 4, 5, and 6 July 2024, at the Shanghai Science and Technology Museum in the city of Shanghai on the east coast of China. Of the entire visitor population over those three days, 23.1% used the app, and all users were invited to give feedback. Of the 83 who did, 56 filled out a short questionnaire that sought answers most pertinent to the museum (whether they learned anything new), to the developer (whether the app failed), and to the research team (whether and how the app altered visits). Additionally, 27 agreed to a full interview, which elaborated on user experience and further questions.

Below, we summarize the most productive themes seen in the data from our 83 participants, leading to four findings in our Discussion topics.

4.1. Enhancement of Physical and Virtual Souvenirs

Our hybrid gifts with AR show significant advantages over both physical and virtual gifts, particularly in ownership, excitement and attachment, duration, interaction, and commemorative significance. Interviewees rated these five aspects across the three types of gifts. Physical gifts provided a strong sense of ownership, excitement, and commemorative significance but lacked durability and interactivity. Digital gifts were durable but scored lower on the other four aspects. The hybrid gifts, however, successfully combined the strengths of both physical and digital souvenirs.

Figure 6. A comparison of three kinds of souvenirs.

Figure 6. A comparison of three kinds of souvenirs.

After reviewing the souvenirs at the museum store, we found that most are models of cars, spacecraft, and dinosaurs. These items, often featuring the museum’s logo, are popular but easily available elsewhere, making them less unique. Traditional souvenirs, while nostalgic, lack the distinctiveness needed to truly represent the museum experience.

In contrast, the AR firework souvenir provides a deeper connection to the exhibits. One visitor shared, “This AR firework reminds me of the minerals exhibition and chemistry knowledge. I might even show it to my family during the Spring Festival.” The AR model effectively evokes exhibition memories, and unlike traditional, fragile souvenirs, it remains durable as long as the server and database are stable.

Souvenirs are often given as gifts, but digital ones lack the ownership and attachment that physical gifts offer. As one interviewee noted, “A digital postcard doesn’t feel as valuable as a physical one—it’s more like sharing than giving.” Our AR souvenir addresses this by combining digital gifts with interactive physical elements, enhancing their value.

Participants appreciated the effort involved in preparing AR gifts, which is often missing in current digital souvenirs. One user mentioned, “I can design my firework show in every detail, making it more real and interactive.” On average, users created 5.47 personalized fireworks and 2.35 overall models, each unique, and recipients recognized and valued the effort put into these personalized gifts.

4.2. Stimulate Learning Motivation

We questioned whether the virtual AR experiments in our app could improve learning experiences and outcomes. One participant noted, “I’m surprised at how much chemistry and physics are in fireworks. The virtual experiment is never boring in such a fascinating setting.” The fun of creating unique fireworks significantly boosted their learning motivation. Another participant mentioned that these virtual AR experiments “brought fun and great memories,” making it easier to recall knowledge. This indicates that combining virtual AR experiments with an edutainment approach helps knowledge stick longer.

Moreover, 82% of participants felt that virtual simulations are crucial in teaching reform due to their ease of use, lower cost, and ability to repeat experiments without time or space constraints. Additionally, 91% found our AR experiments more engaging than traditional simulations and were eager to use them more, highlighting their potential to enhance learning motivation and retention.

4.3. Distinctive Self-Making Experience Leads to Long-Term Memory

When asked about their feelings while creating fireworks and gifts, one participant described the process as “an art feast”, while another eagerly anticipated their friend’s reaction, which likely contributed to the “excitement” some felt during the design.

Backend data revealed that nearly 18% of uploaded fireworks were picture fireworks—generated from user images—while 24% were system presets. Participants appreciated the variety of default picture fireworks, with one praising their thematic diversity and another enjoying the creative process of combining them. Some users, however, expressed frustration over the inability to resize images. Overall, the picture fireworks feature provided significant creative freedom and enhanced the joy of creation.

Moreover, the app’s feature of recording personalized messages with gifts proved memorable. Research indicates that emotionally engaging experiences are more likely to be retained in long-term memory. To test this, we conducted an experiment with 30 adolescents (ages 13-16), testing each participant in two sessions one week apart. They were given ten gifts with recorded messages and ten without, then asked to recall the content. Using signal detection theory, we calculated sensitivity (d-prime) to measure memory recall. As shown in Figure 7, d’ was calculated by subtracting the z-score for the false alarm rate from the z-score for the hit rate: d’ = z(hit rate) – z(false alarm rate).

Figure 7. D prime as a function of mode and time.

Figure 7. D prime as a function of mode and time.

4.4. Framing the Museum Visit

To study the impact of AR on the exhibition viewing experience, we conducted user tests and interviews at its peak in the adolescent stage (13-16), which is the fastest and highest capacity of memory in life.

Table 2. Experiment description.

Purpose of the experiment	Compare the impact of AR on the viewer
Subjects’ characteristics	Middle school students in the 13-16 age group*
Experimental method	Control experiment (Group A: control group; Group B: experimental group)
Sample size	Total 20 persons (Group A: 10 persons; Group B: 10 persons)
Experiment description	The permanent exhibition area of “Earth Home” was chosen as the experimental exhibition gallery. Two groups of junior high school students were arranged to visit the display area, one group visited the original one and the other group visited the one under AR fireworks design service, and the duration of the visit was limited to 30 minutes to compare the breadth and depth of memory of the exhibits in the display area between the two groups.

Table 2. Experiment description.

Purpose of the experiment	Compare the impact of AR on the viewer
Subjects’ characteristics	Middle school students in the 13-16 age group*
Experimental method	Control experiment (Group A: control group; Group B: experimental group)
Sample size	Total 20 persons (Group A: 10 persons; Group B: 10 persons)
Experiment description	The permanent exhibition area of “Earth Home” was chosen as the experimental exhibition gallery. Two groups of junior high school students were arranged to visit the display area, one group visited the original one and the other group visited the one under AR fireworks design service, and the duration of the visit was limited to 30 minutes to compare the breadth and depth of memory of the exhibits in the display area between the two groups.

We print cards that cover the key knowledge points and exhibit content covered in the exhibition hall, as well as some vague distractions that do not appear in the hall. We asked adolescents who completed the exhibition visit to choose whether the card contents appeared in the exhibition hall and to recall the corresponding contents of the selected cards. The statistical results of the experiment were used to examine the effect of AR technology on the memory of the exhibition viewers’ visit. The exact card content represents the memory of learning points; the exact number of cards represents the breadth of memory of learning points; and the number of recalled cards corresponding to the content represents the depth of memory of learning points.

Figure 8. Breadth and depth of learning effect.

Figure 8. Breadth and depth of learning effect.

Statistics show that the AR experiment significantly enhanced subjects’ impressions of specific exhibits. Compared to viewing the original exhibition without AR, the experience with AR guidance proved more effective. Additionally, combining AR technology with junior high school physics and chemistry textbooks helped students better understand and retain key learning points.

To demonstrate that the hybrid souvenir maintains the connection between the viewer and the exhibition, we tracked data from users who revisited the app within six months after their initial visit. The data was visualized in a chart, where each color represents a different user. Users are marked with dots each time they open the app—the larger the dot, the longer the session, and the more interactions, the denser the cluster of dots.

According to backend statistics, nearly 83% of users reopened the app for review within 30 days after the exhibition ended, and about 35% used the app two or more times within six months, with each session averaging over 20 minutes.

Figure 9. Frequency of App use after users leave the museum.

Figure 9. Frequency of App use after users leave the museum.

As the primary purchasers, parents can buy souvenirs for their children while also satisfying their own emotional needs through the added value of the souvenirs, such as virtual elements that combine fireworks, voice, and photos accessible via QR codes. Traditional science and technology museum souvenirs often focus on generating revenue and are typically aimed at younger students. These items, often unrelated to the museum or simply featuring a logo, fail to resonate with parents, who may see them as unnecessary clutter. This poor consumer experience can discourage parents from buying souvenirs altogether. By allowing parents to upload personal photos and voice recordings, the souvenir design enhances the emotional value, making the purchase more meaningful and satisfying.

The design also intelligently segments visitors by age and needs, planning viewing paths that help disperse crowds during science education activities. Traditional museum events often lead to crowding around specific exhibits, creating a poor experience for both participants and other visitors. By using a mobile platform and age-based path design, we were able to better distribute participants, reducing crowd density and improving the overall visitor experience.

5. Discussion

5.1. AR Personalization

Researchers have explored personalized products in museums, highlighting the potential of personalized gifts to deliver unique experiences [4]. Spence et al. proposed a bottom-up personalization mechanism where visitors select and share artworks [5]. However, relying solely on photos or sounds for souvenirs does not fully meet the growing demand for personalization.

We propose AR souvenirs as a promising customization solution. AR souvenirs allow for greater creativity, particularly in model design. In our application, models are categorized as predefined or DIY, with the latter generated using custom parameters and images to add artistry. Audio files can be linked to these models, enabling a more concrete personalization experience compared to abstract alternatives. More research is needed to develop richer, more artistic AR models for customization.

AR souvenirs also present new opportunities in the gift and packaging market. Using established AR image recognition, recognizable markers on packaging can trigger the display of corresponding models on smartphones, enhancing traditional gifts with a vivid, social dimension. Although our current hybrid souvenir—a gift box with a recognizable picture and QR code—may lack functionality, linking AR models to real objects could increase their appeal. For example, scanning a globe souvenir could reveal a museum’s AR installation and past interactions with it. As object recognition technology improves, AR could become a compelling alternative to traditional photos. Our research suggests that DIY AR souvenirs can leave a lasting impression on visitors.

However, challenges remain, particularly with webAR technology, custom content design, and object recognition. Many gift recipients are unwilling to download an app to view souvenirs due to inconvenience. While webAR offers a quick solution, its interaction capabilities are limited, and Unity lacks suitable webAR plugins. Further research is necessary to improve the functionality and usability of webAR.

5.2. Focus on Accessibility and Usability

Science and technology museums increasingly incorporate AR and VR to enhance visitor experiences, but our focus is on accessibility and usability rather than just novel features. The primary visitors to these museums are primary and secondary school students. Traditional scientific devices, the main competitors, remain popular with children due to their immediate feedback and accessibility. These museums play a crucial role in broadening children’s horizons, sparking their interest in science, and helping them learn through engaging interactions and visuals. While children may not fully grasp the scientific concepts at the time, they are often captivated by the design and later seek explanations from exhibits or their parents. Even if the principles aren’t immediately understood, the experience may leave a lasting impression that aids future learning.

New technology devices or applications often come with higher learning curves and more complex processes than traditional ones. Children may lose interest if they encounter obstacles or if the process is too long and lacks immediate feedback. Therefore, designers should prioritize clear feedback at every step and simplify the process to maintain engagement.

6. Conclusion

In conclusion, our exploration of the AR firework souvenir and the application of the STEAM education approach in the science and technology museum was highly successful, revealing numerous possibilities for future research. The STEAM and AR-based education model effectively captured children’s interest in science. As the first study to propose AR hybrid souvenirs, we see significant potential in future markets for packaging, souvenirs, and gifts. However, further research on usability and accessibility is needed. Additionally, we optimized parental involvement in the museum, addressing issues related to crowd management.