Submitted:
28 May 2025
Posted:
04 June 2025
You are already at the latest version
Abstract
Keywords:
1. Introduction
1.1. Results from ReInHerit Research on Digital Tools
1.2. Demographics
1.3. Visitor Preferences
1.4. Heritage Professionals Needs
- 67.33% of museums and cultural heritage sites rely on standard ICT tools, while only 33% use innovative ICT tools. This highlights a need for integrating more innovative tools into the sector to improve visitor engagement and experience.
- The analysis revealed that smaller organizations are more likely to rely on standard ICT tools and face greater challenges in adopting innovative solutions. This underscores the need for sharing digital platform that can support museums of all sizes, offering tailored solutions to their specific needs.
- AI and gamification tools (e.g., chatbots, digital storytelling) were identified as important but rarely used. These tools can enhance visitor interaction and engagement, making them crucial for future development.
- Human Resources: Most organizations do not employ dedicated professionals for technological implementation. Instead, they rely on third-party consultants or lack the resources to develop digital tools internally. This indicates a need for training and upskilling heritage professionals to become active agents in the digital transformation of cultural heritage institutions (Figure 9).
2. Materials and Methods
2.1. Insights on AI and Museums
2.2. The ReInherit Toolkit Method
3. Results
3.1. Strike-a-Pose
3.2. Face-Fit
- To implement these experiences as challenges that enhance visitor engagement and provide personalized takeaways of the visit, encouraging post-visit exploration.
- To generate user-created content that can amplify engagement on social media platforms.
- To employ advanced AI methods optimized for mobile execution, supporting a BYOD strategy for widespread accessibility.
3.3. Using CLIP for Artwork Recognition and Image Retrieval
3.4. Smart Lens
- Content-based Image Retrieval (CBIR) - This method compares visual descriptors from the user’s live camera feed with those extracted from a curated image dataset. Each artwork is not only represented as a whole but also partitioned into segments, so that fine-grained features can be detected and matched efficiently.
- Classification - A neural network model, specifically fine-tuned for the collection, assigns a class label to the input frame based on overall appearance. The recognition result is accepted only if the confidence score exceeds a designated threshold. This lightweight solution is ideal for running directly on mobile devices.
- Object Detection - In this mode, the system pinpoints and labels multiple details within a single artwork using bounding boxes. The underlying model, optimized for detecting artwork-specific elements, selects only those regions whose confidence level meets predefined criteria. This approach is particularly suited for complex works with multiple visual components.
3.5. Multimedia Chatbot: VIOLA
- A neural network classifies the user’s query, determining whether it pertains to the visual content or the contextual aspects of the artwork.
- A question-answering (QA) neural network uses contextual information about the artwork, stored in JSON format, to address questions related to its context.
- A visual question-answering (VQA) neural network processes the visual data and the visual description of the artwork, stored in JSON format, to answer questions about the content of the artworkn[53].
3.6. Smart Video and Photo Restorer
| Method | PSNR ↑ | SSIM ↑ | LPIPS ↓ |
|---|---|---|---|
| DeOldify [61] | 11.56 | 0.451 | 0.671 |
| Ours | 34.78 | 0.939 | 0.063 |
4. Discussion
5. Conclusions
Author Contributions
Funding
Acknowledgments
Conflicts of Interest
Abbreviations
| CV | Computer Vision |
| AI | Artificial Intelligence |
| CH | Cultural Heritage |
| ICT | Information and Communication Technologies |
| GDPR | General Data Protection Regulation |
| BYOD | Bring Your Own Device |
| API | Application Programming Interface |
| CLIP | Contrastive Language-Image Pre-Training |
| CBIR | Content-based Image Retrieval |
| GPU | Graphics Processing Unit |
| VQA | Visual Question Answering |
| PSNR | Peak signal-to-noise ratio |
| SSIM | Structural Similarity Index |
| LPIPS | Learned Perceptual Image Patch Similarity |
References
- Stamatoudi, I.; Roussos, K. A Sustainable Model of Cultural Heritage Management for Museums and Cultural Heritage Institutions. ACM Journal on Computing and Cultural Heritage, 2024. [CrossRef]
- Gaia, G.; Boiano, S.; Borda, A., Engaging Museum Visitors with AI: The Case of Chatbots. In Museums and Digital Culture: New Perspectives and Research; Giannini, T.; Bowen, J.P., Eds.; Springer International Publishing: Cham, 2019; pp. 309–329. [CrossRef]
- Adahl, S.; Träskman, T. Archeologies of Future Heritage: Cultural Heritage, Research Creations, and Youth. Mimesis Journal 2024, 13, 487–497. [Google Scholar] [CrossRef]
- Pietroni, E. Multisensory Museums, Hybrid Realities, Narration, and Technological Innovation: A Discussion Around New Perspectives in Experience Design and Sense of Authenticity. Heritage 2025, 8. [Google Scholar] [CrossRef]
- Ivanov, R.; Velkova, V. Analyzing Visitor Behavior to Enhance Personalized Experiences in Smart Museums: A Systematic Literature Review. Computers 2025, 14. [Google Scholar] [CrossRef]
- MusuemBooster. Museum Innovation Barometer, 2022. https://www.museumbooster.com/mib. Accessed on 7 May 2025.
- Giannini, T.; Bowen, J. Museums and Digital Culture: From Reality to Digitality in the Age of Covid-19, 2021. [CrossRef]
- Thiel, S.; Bernhardt, J.C. (Eds.) Reflections, Perspectives and Applications; transcript Verlag: Bielefeld, 2023; pp. 117–130. [Google Scholar] [CrossRef]
- Clarencia, E.; Tiranda, T.G.; Achmad, S.; Sutoyo, R. The Impact of Artificial Intelligence in the Creative Industries: Design and Editing. In Proceedings of the Proc. of International Seminar on Application for Technology of Information and Communication (iSemantic); 2024; pp. 440–444. [Google Scholar] [CrossRef]
- Muto, V.; Luongo, S.; Sepe, F.; Prisco, A. Enhancing Visitors’ Digital Experience in Museums through Artificial Intelligence. In Proceedings of the Proc. of Business Systems Laboratory International Symposium, Palermo, Italy; 2024. [Google Scholar]
- Furferi, R.; Di Angelo, L.; et al. Enhancing traditional museum fruition: Current state and emerging tendencies. Heritage Science 2024, 12. [Google Scholar] [CrossRef]
- Villaespesa, E.; Murphy, O. This is not an apple! Benefits and challenges of applying computer vision to museum collections. Museum Management and Curatorship 2021, 36, 362–383. [Google Scholar] [CrossRef]
- Osoba, O.A.; Welser IV, W.; Welser, W. An intelligence in our image: The risks of bias and errors in artificial intelligence; Rand Corporation, 2017.
- Villaespesa, E.; Murphy, O. THE MUSEUMS + AI NETWORK - AI: A Museum Planning Toolkit. https://themuseumsai.network/toolkit/, 2020. Accessed on 7 May 2025. [CrossRef]
- Zielke, T. Is Artificial Intelligence Ready for Standardization? In Proceedings of the Systems, Software and Services Process Improvement; Yilmaz, M.; Niemann, J.; Clarke, P.; Messnarz, R., Eds., Cham; 2020; pp. 259–274. [Google Scholar]
- Pasikowska-Schnass, M.; Young-Shin, L. Members’ Research Service, “Artificial intelligence in the context of cultural heritage and museums: Complex challenges and new opportunities” EPRS | European Parliamentary Research PE 747.120 – May 2023. https://www.europarl.europa.eu/RegData/etudes/BRIE/2023/747120/EPRS_BRI(2023)747120_EN.pdf. Accessed on 7 May 2025.
- CM. Recommendation CM/Rec(2022)15 Adopted by the Committee of Ministers on 20 May 2022 at the 132nd Session of the Committee of Ministers. https://search.coe.int/cm?i=0900001680a67952, 2022. Accessed on 7 May 2025.
- Furferi, R.; Colombini, M.P.; Seymour, K.; Pelagotti, A.; Gherardini, F. The Future of Heritage Science and Technologies: Papers from Florence Heri-Tech 2022. Heritage Science 2024, 12, 155. [Google Scholar] [CrossRef]
- Boiano, S.; Borda, A.; Gaia, G.; et al. Ethical AI and Museums: Challenges and New Directions. In Proceedings of the Proc. of EVA London 2024, 2024. [Google Scholar] [CrossRef]
- Orlandi, S.D.; De Angelis, D.; Giardini, G.; Manasse, C.; Marras, A.M.; Bolioli, A.; Rota, M. IA FAQ INTELLIGENZA ARTIFICIALE - AI FAQ ARTIFICIAL INTELLIGENCE. https://doi.org/10.5281/zenodo.15069460, 2025. Zenodo. Accessed on 7 May 2025.
- Nikolaou, P. Museums and the Post-Digital: Revisiting Challenges in the Digital Transformation of Museums. Heritage 2024, 7, 1784–1800. [Google Scholar] [CrossRef]
- Barekyan, K.; Peter, L. Digital Learning and Education in Museums: Innovative Approaches and Insights. https://www.ne-mo.org/fileadmin/Dateien/public/Publications/NEMO_Working_Group_LEM_Report_Digital_Learning_and_Education_in_Museums_12.2022.pdf, 2023 NEMO. Accessed on 7 May 2025.
- Falk, J.H.; Dierking, L.D. The Museum Experience, 1st ed.; Routledge, 2011. [CrossRef]
- Falk, J.; Dierking, L. The Museum Experience Revisited; 2016. [CrossRef]
- Mazzanti, P.; Sani, M. Emotions and Learning in Museums. https://www.ne-mo.org/fileadmin/Dateien/public/Publications/NEMO_Emotions_and_Learning_in_Museums_WG-LEM_02.2021.pdf, 2021 NEMO. Accessed on 7 May 2025.
- Damásio, A.R. Descartes’ Error: Emotion, Reason, and the Human Brain; Avon Books: New York, NY, USA, 1994. [Google Scholar]
- Barrett, L.F. How Emotions Are Made; Houghton Mifflin Harcourt: New York, NY, USA, 2017. [Google Scholar]
- Pietroni, E.; Pagano, A.; Fanini, B. UX Designer and Software Developer at the Mirror: Assessing Sensory Immersion and Emotional Involvement in Virtual Museums. Studies in Digital Heritage 2018, 2, 13–41. [Google Scholar] [CrossRef]
- Pescarin, S.; Città, G.; Spotti, S. Authenticity in Interactive Experiences. Heritage 2024, 7, 6213–6242. [Google Scholar] [CrossRef]
- Lieto, A.; Striani, M.; Gena, C.; Dolza, E.; Anna Maria, M.; Pozzato, G.L.; Damiano, R. A sensemaking system for grouping and suggesting stories from multiple affective viewpoints in museums. Human–Computer Interaction 2024, 39, 109–143. [Google Scholar] [CrossRef]
- Damiano, R.; Lombardo, V.; Monticone, G.; Pizzo, A. Studying and designing emotions in live interactions with the audience. Multimedia Tools and Applications 2021, 80, 6711–6736. [Google Scholar] [CrossRef]
- Khan, I.; Melro, A.; Amaro, A.C.; Oliveira, L., Role of Gamification in Cultural Heritage Dissemination: A Systematic Review; 2021; pp. 393–400. https://doi.org/10.1007/978-981-16-2377-6_37. [CrossRef]
- Alexander, J. Gallery One at the Cleveland Museum of Art. Curator: The Museum Journal 2014, 57, 347–362. [Google Scholar] [CrossRef]
- Alexander, J.; Barton, J.; Goeser, C. Transforming the Art Museum Experience: Gallery One. In Proceedings of the Museums and the Web 2013; Proctor, N.; Cherry, R., Eds., Silver Spring, MD; 2013. [Google Scholar]
- Fan, S.; Wei, J. Enhancing Art History Education Through the Application of Multimedia Devices. In Proceedings of the Proc. of International Conference on Internet, Education and Information Technology (IEIT 2024). Atlantis Press, 2024, pp. 425–436. [CrossRef]
- Münster, S.; Maiwald, F.; di Lenardo, I.; Henriksson, J.; Isaac, A.; Graf, M.M.; Beck, C.; Oomen, J. Artificial Intelligence for Digital Heritage Innovation: Setting up a R&D Agenda for Europe. Heritage 2024, 7, 794–816. [Google Scholar] [CrossRef]
- Fiorucci, M.; Khoroshiltseva, M.; Pontil, M.; Traviglia, A.; Del Bue, A.; James, S. Machine Learning for Cultural Heritage: A Survey. Pattern Recognition Letters 2020, 133, 102–108. [Google Scholar] [CrossRef]
- Pansoni, S.; Tiribelli, S.; Paolanti, M.; Di Stefano, F.; Frontoni, E.; Malinverni, E.S.; Giovanola, B.; et al. Artificial intelligence and cultural heritage: Design and assessment of an ethical framework. International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences 2023, 48, 1149–1155. [Google Scholar] [CrossRef]
- UNESCO. Guidance for generative AI in education and research, 2023. Accessed on 7 May 2025. [CrossRef]
- UNESCO. Recommendation on the Ethics of Artificial Intelligence SHS/BIO/PI/2021/1. https://unesdoc.unesco.org/ark:/48223/pf0000381137. Accessed on 7 May 2025.
- Rachabathuni, P.K.; Mazzanti, P.; Principi, F.; Ferracani, A.; Bertini, M. Computer Vision and AI Tools for Enhancing User Experience in the Cultural Heritage Domain. In Proceedings of the Proc. of HCI International 2024 – Late Breaking Papers; Zaphiris, P.; Ioannou, A.; Sottilare, R.A.; Schwarz, J.; Rauterberg, M., Eds., Cham; 2025; pp. 345–354. [Google Scholar] [CrossRef]
- Donadio, M.G.; Principi, F.; Ferracani, A.; Bertini, M.; Del Bimbo, A. Engaging Museum Visitors with Gamification of Body and Facial Expressions. In Proceedings of the Proc. of ACM International Conference on Multimedia, New York, NY, USA, 2022; MM ’22; pp. 7000–7002. [Google Scholar] [CrossRef]
- Nielsen, J. Iterative user-interface design. Computer 1993, 26, 32–41. [Google Scholar] [CrossRef]
- Kartynnik, Y.; Ablavatski, A.; Grishchenko, I.; Grundmann, M. Real-time Facial Surface Geometry from Monocular Video on Mobile GPUs. https://arxiv.org/abs/1907.06724, 2019, [arXiv:cs.CV/1907.06724].
- Ekman, P.; Friesen, W.V.; O’sullivan, M.; Chan, A.; Diacoyanni-Tarlatzis, I.; Heider, K.; Krause, R.; LeCompte, W.A.; Pitcairn, T.; Ricci-Bitti, P.E.; et al. Universals and cultural differences in the judgments of facial expressions of emotion. Journal of personality and social psychology 1987, 53, 712. [Google Scholar] [CrossRef]
- Reinhard, E.; Adhikhmin, M.; Gooch, B.; Shirley, P. Color transfer between images. IEEE Computer graphics and applications 2001, 21, 34–41. [Google Scholar] [CrossRef]
- Gallese, V. Embodied Simulation. Its Bearing on Aesthetic Experience and the Dialogue Between Neuroscience and the Humanities. Gestalt Theory 2019, 41, 113–127. [Google Scholar] [CrossRef]
- Karahan, S.; Gül, L.F. Mapping Current Trends on Gamification of Cultural Heritage. In Proceedings of the Game + Design Education; Cordan, Ö.; Dinçay, D.A.; Yurdakul Toker, Ç.; Öksüz, E.B.; Semizoğlu, S., Eds., Cham; 2021; pp. 281–293. [Google Scholar]
- Bonacini, E.; and, S.C.G. Gamification and cultural institutions in cultural heritage promotion: A successful example from Italy. Cultural Trends 2022, 31, 3–22. [Google Scholar] [CrossRef]
- OPEN-AI. CLIP: Connecting text and images. https://openai.com/research/clip, 2021. Accessed on 7 May 2025.
- Chiaro, R.D.; Bagdanov, A.; Bimbo, A.D. NoisyArt: A Dataset for Webly-supervised Artwork Recognition. In Proceedings of the Proc. of International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2019). INSTICC, SciTePress; 2019; pp. 467–475. [Google Scholar] [CrossRef]
- Baldrati, A.; Bertini, M.; Uricchio, T.; Del Bimbo, A. , Exploiting CLIP-Based Multi-modal Approach for Artwork Classification and Retrieval. In The Future of Heritage Science and Technologies: ICT and Digital Heritage; Springer International Publishing, 2022; pp. 140–149. [CrossRef]
- Bongini, P.; Becattini, F.; Bimbo, A.D. Is GPT-3 all you need for Visual Question Answering in Cultural Heritage? https://arxiv.org/abs/2207.12101, 2023, [arXiv:cs.CV/2207.12101].
- Rachabatuni, P.K.; Principi, F.; Mazzanti, P.; Bertini, M. Context-aware chatbot using MLLMs for Cultural Heritage. In Proceedings of the Proc. of ACM Multimedia Systems Conference, New York, NY, USA, 2024; MMSys ’24; pp. 459–463. [Google Scholar] [CrossRef]
- Boiano, S.; Borda, A.; Gaia, G.; Rossi, S.; Cuomo, P. Chatbots and new audience opportunities for museums and heritage organisations. In Proceedings of the Proc. of Conference on Electronic Visualisation and the Arts, Swindon, GBR, 2018; EVA ’18; pp. 164–171. [Google Scholar] [CrossRef]
- Gustke, O.; Schaffer, S.; Ruß, A. CHIM—Chatbot in the Museum. Exploring and Explaining Museum Objects with Speech-Based AI. In AI in Museums; Thiel, S.; Bernhardt, J.C., Eds.; transcript Verlag: Bielefeld, 2023; pp. 257–264. Online ISBN: 978-3-8394-6710-7, Creative Commons Attribution 4.0 License (CC BY 4.0), Retrieved 2025-01-18 15:14:07-CET, ISBN 978-3-8394-6710-7. [CrossRef]
- Galteri, L.; Bertini, M.; Seidenari, L.; Uricchio, T.; Del Bimbo, A. Increasing Video Perceptual Quality with GANs and Semantic Coding. In Proceedings of the Proc. of ACM International Conference on Multimedia, New York, NY, USA, 2020; MM ’20; pp. 862–870. [Google Scholar] [CrossRef]
- Simonyan, K.; Zisserman, A. Very Deep Convolutional Networks for Large-Scale Image Recognition. In Proceedings of the Proc. of International Conference on Learning Representations, ICLR; Bengio, Y.; LeCun, Y., Eds. 2015. [Google Scholar]
- Wang, Z.; Bovik, A.C.; Sheikh, H.R.; Simoncelli, E.P. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing 2004, 13, 600–612. [Google Scholar] [CrossRef] [PubMed]
- Zhang, R.; Isola, P.; Efros, A.A.; Shechtman, E.; Wang, O. The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. In Proceedings of the Proc. of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Los Alamitos, CA, USA; 2018; pp. 586–595. [Google Scholar] [CrossRef]
- Antic. DeOldify. https://github.com/jantic/DeOldify. Accessed on 7 May 2025.
- Murphy, O.; Villaespesa, E.; Gaia, G.; Boiano, S.; Elliott, L. MUSEI E INTELLIGENZA ARTIFICIALE: UN TOOLKIT DI PROGETTAZIONE; 2024. [CrossRef]
| 1 | The Hub collects resources, and training material to foster and support cultural tourism in museums and heritage sites, and a networking platform to connect and exchange experiences. Website: https://reinherit-hub.eu/resources
|
| 2 | AI-Based Toolkit for Museums and Cultural Heritage Siteshttps://www.europeanheritagehub.eu/document/ai-based-toolkit-for-museums-and-cultural-heritage-sites/
|
| 3 | ReInHerit D3.9 - Training Curriculum and Syllabi - https://ucarecdn.com/095df394-fad6-4f35-bdcc-09 769 931d0b8dd2/
|
| 4 | |
| 5 | D3.1 - ReInHerit National Surveys Report - https://ucarecdn.com/54faa991-1570-4a53-9e8a-c1dea0a33110/
|
| 6 | The national surveys complied with GDPR, ensuring full anonymization of data by not requesting personal details. Participants were also informed about the survey’s purpose and how the data would be used. Similarly, the focus groups followed strict ethical guidelines, with informed consent procedures and data anonymization carefully planned according to the project’s Ethics and Data Management Plan. An additional central aim was to ensure full adherence to GDPR standards. See ReInHerit Deliverables D2.1, D2.4, D3.1 https://reinherit-hub.eu/deliverables
|
| 7 | D3.2 - ReInHerit Toolkit Strategy Report - https://ucarecdn.com/71ffe888-3c0d-470d-962d-ab145edcff3f/
|
| 8 | |
| 9 | |
| 10 | Ethical Aspects and Scientific Accuracy of AI/CV-based tools: https://reinherit-hub.eu/bestpractices/db1bd5ab-218f-480b-b709-06ac9ab72b33
|
| 11 | ReInHerit Applications https://reinherit-hub.eu/applications/
|
| 12 | |
| 13 | |
| 14 | |
| 15 | |
| 16 | |
| 17 | |
| 18 | |
| 19 | |
| 20 | Students and researchers from different international academic backgrounds participated in the international XR/AI Summer School 2023 from 17 to 22 July 2023 in Matera Italy, working on the topics of Extended Reality and Artificial Intelligence. More info on ReInHerit Hackathon and project proposals:https://reinherit-hub.eu/summerschool/
|
| 21 | AI-Based Toolkit for Museums and Cultural Heritage Sites https://www.europeanheritagehub.eu/document/ai-based-toolkit-for-museums-and-cultural-heritage-sites/
|
























| Topic | Description |
|---|---|
| Playful Experience | AI and CV tools are applied to foster learning and build a deeper connection between visitors and artworks. Interactions and gamified experiences are designed to trigger emotion, encourage creativity, and support participatory engagement. |
| New Audience | Younger audiences, who tend to be more familiar with digital technologies, are a key target of the ReInherit Toolkit, which aims to increase their active participation in museum experiences. |
| Sustainability | Smaller museums often lack the resources and skills to adopt digital tools, making training and capacity-building crucial for effective heritage innovation. |
| Bottom-Up | The development process follows a community-driven model, where local participants are actively involved through workshops and hackathons. This inclusive method ensures that the tools reflect the needs and insights of the users themselves. |
| Co-Creation | The innovative goal is to offer not just a tool as a final product, but a collaborative development process that fosters mediation between different disciplinary sectors. |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2025 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
