From Data to Impact: Assessing the Value of Cultural Heritage in the Digital Age

1. Introduction

This article provides an overview of the current situation regarding the digital transformation of the heritage sector and examines the challenges and opportunities associated with assessing the impact of digital cultural heritage initiatives in Europe. When taking stock of academic research concerning digital culture and heritage, it becomes visible that this is a complex notion not conforming to a singular definition that would encompass all its facets and provide comprehensive reasons why it is important. Recent academic research has, among other issues, delved into various aspects of digital cultural heritage, including digital humanities and digital heritage in AI environment [1], linked data and cultural heritage [2], cultural heritage data from the perspective of digital humanities [3], crowdsourcing and heritage [4], cultural heritage on social media [5], etc. Lian and Xie point out that the main current themes concerning digital heritage related research cover issues of relevant technological innovation and its application, information management and technical support, and digitization and preservation of cultural heritage [6]. While such studies highlight the use and potential of digital tools in cultural heritage, the specific impact of digital cultural heritage projects on broader cultural, social or economic contexts remains a relatively unexplored area [7].

However, the evolving role of cultural heritage institutions nowadays is closely tied to their ability to create a meaningful impact on society. Clearly communicating the value of digital collections and demonstrating their impact to decision-makers, funders and users is becoming essential to justify current activities and secure continued support for their development and upkeep [8]. Equally important is that performing internal self-evaluations should allow heritage institutions to check if they are on the right track. Yet, the question remains: How can the success of digital cultural heritage resources and projects be measured, and against which underpinning values should they be evaluated?

In approaching this subject, we consider that culture is “collective memory, dependent on communication for its creation, extension, evolution and preservation” [9] (p.19). This implies that whenever we discuss culture, communication is always implicit, as our collective knowledge has always been communicated and preserved through the existing cultural communication structures with technologies playing a crucial role in enabling and facilitating those processes [10]. In other words, the ability to acquire, share, and innovate knowledge is essential for the preservation and development of any culture [11]. The digital context has paved the way for numerous collaborative and creative processes, with data now serving as a valuable resource for education, arts, etc., while enabling us to share and preserve our cultural memory. Vast amounts of data about human society and culture, both present and past, have become a new frontier for digital initiatives in institutions worldwide. This is the reason why we consider digital heritage to be truly important and where we find its value – in considering it as knowledge resource that should be used in the development of creativity and creating an enabling environment for empowering citizens.

In recent decades, digital culture and digital cultural heritage has gained a more prominent place in the political agenda of the European Union (EU). EU considers cultural heritage as a strategic resource for sustainable development, and this encompasses digital heritage as well. Ambitiously, the European Commission [12] (p. 8) set the goals for the digital future: “by 2030, Member States should digitise in 3D all monuments and sites that are considered to be cultural heritage at risk, and 50% of the most physically visited cultural and heritage monuments, buildings and sites.” The official EU discourse has shifted from focusing on “preservation” and “access” to emphasizing “digital transformation”. The digitization of cultural heritage has been seen as central to transforming it into new knowledge resources, unlocking opportunities for the development of new services and content [13]. Moreover, the potential of digital cultural heritage to drive job creation across various economic sectors has been placed in the focus of policymakers, and expectations have been raised that providing access to and enabling the reuse of digital content can generate additional revenue streams for cultural heritage institutions [14].

However, ensuring that digitized content reaches its intended audience is a challenging endeavor. In practice, heritage institutions strive to balance their missions of preserving and providing access to our shared heritage with the opportunities and challenges brought by the digital age, emphasizing the significance of their collections by digitizing them for both preservation and accessibility [11]. Cultural heritage institutions are increasing their digital offers with an aim to enhance user experience and attract new audiences. It is evident that participatory practices have become increasingly valuable in the digital heritage sector, transforming public engagement with cultural heritage institutions [15].

2. Materials and Methods

This paper explores the complex landscape of impact assessment (IA) for digital cultural heritage initiatives within the European Union. Based on a desk research methodology, the research provides a systematic review encompassing: 1) relevant EU-funded projects focused on (digital) heritage impact assessment, with particular attention paid to their outputs such as reports, guidelines, frameworks, and toolkits; 2) relevant academic literature on the subject of impact assessment for digital cultural heritage, identified through a multi-faceted search approach using traditional academic databases, the AI-powered research platform scite.ai, and Google Scholar, with keywords including “impact of digital cultural heritage”, “evaluation of digital cultural heritage” and “impact assessment framework for cultural heritage”; and 3) the current EU policy framework related to digital cultural politics, and the digital transformation of cultural heritage institutions, including key initiatives like Europeana and the Data Space for Cultural Heritage.

This three-pronged approach aimed to ensure a comprehensive and contextualized understanding of the challenges and opportunities associated with measuring the impact of digital cultural heritage within the evolving European policy landscape. Furthermore, during the research on impact assessment frameworks, several other crucial and interconnected concepts emerged, including digital maturity, accountability, resilience, and sustainability. These concepts were considered essential to the broader understanding of the relevance of impact assessment and, consequently, were further explored through a combination of EU policy documents and academic literature to establish their interrelationships within the context of digital cultural heritage.

A critical component of the research involves a comparative analysis of six impact assessment frameworks, including the Balanced Value Impact Model (BVI), the Europeana Impact Playbook, the Change Impact Assessment Framework, Impactomatrix, the MOI Framework, and the SoPHIA Model. These frameworks are evaluated based on their proposed methodologies, specific thematic focus, and the inclusion (or lack thereof) of proposed indicators.

A critical lens was applied throughout the analysis, allowing for a nuanced understanding of the interplay between policy, practice, and conceptual IA frameworks. The analysis focuses on the strengths and limitations of each framework and assesses their suitability for the current capacities and needs of cultural heritage institutions. The insights gained through the reviewed literature, project outputs, and the analysed IA frameworks enabled us to identify key challenges and propose directions for developing more standardized and holistic approaches to impact assessment for digital cultural heritage, emphasizing the need to move beyond data collection to demonstrating meaningful change and long-term sustainability.

3. Results

3.1. Digital Transformation and Data Policies

When the discourse shifted from digitisation and digitalisation to digital transformation, the issue of sustainability became more prominently the focus of both policymakers and heritage practitioners. However, this is not a straightforward issue. Digital transformation goes beyond the operational aspects of cultural heritage institutions; it reshapes their way of thinking. It is not only about technology and resources but also about the people and skills involved [16]. It is about turning digital cultural assets into products and services that make a meaningful impact on society while ensuring the long-term sustainability of digital resources.

Digitalisation implies the preservation of cultural heritage for future generations (cultural memory objective), as well as reducing costs and energy consumption associated with physical visits to cultural heritage sites (environmental sustainability objective), and this is clearly recognised in The Recommendation of 10.11.2021 on a Common European Data Space for Cultural Heritage [12]. Nevertheless, heritage sector needs to be mindful of the fact that digital infrastructure has a significant environmental impact. Digitisation is inextricably linked to augmenting the volume of digital data. As the volume of data exponentially increases, it raises energy consumption due to the needed infrastructure, which has significant ramifications for environmental impact. The recently introduced concept of digital sobriety seeks to advocate for reducing the environmental impact of digital projects. This, among other things, entails minimising energy usage throughout the digitisation process, implementing efficient storage solutions, establishing sustainable practices for digital archive management, etc. [15]. To strike a necessary balance, developing metrics to evaluate the cultural sector’s performance, including its contribution to ecological sustainability is becoming necessity.

Policies play a crucial role in shaping how the cultural sector adapts its communication methods [11,17] and data issues are increasingly becoming the subject of European digitisation policies, influencing the creation, distribution and consumption of digital content. The numerous EU strategies around data such as Open Data Strategy [18], Open Data Directive [19] or the Copyright Directive [20] – which aim to regulate data issues (or content), etc. – are at the very core of cultural sector policies. The convergence of digital and sustainability issues is at the core of European Strategy for Data: “… making more data available and improving the way in which data is used is essential for tackling societal, climate and environment-related challenges, contributing to healthier, more prosperous and more sustainable societies” [21].

The Recommendation of 10.11.2021 on a Common European Data Space for Cultural Heritage [12] highlights the importance of data in cultural heritage and points towards the EU’s future direction. The openness of mediatized memory has been described as offering an alternative memory boom, characterized by an unfinished past and a revitalized future (Lunenfeld 2011 as cited in [22]). Linking policy with practice, platforms like Europeana and DARIAH EU drive digital transformation of the sector, facilitating access, reuse, and sharing of digitized heritage. Europeana, a central access point to European online heritage, is seen as the cornerstone for creating the “common data space” for cultural heritage sector and serves as a platform for reusing and sharing digitized heritage materials. In addition to competence pooling from the heritage sector, transforming cultural content to social and economic assets, and informing EU digital cultural policy, Europeana plays a role in fostering the European citizenship by promoting European identity through its rich repository of cultural content [23]. In 2021, Europeana provided access to 52 million cultural heritage assets, 45% of which could have been reused in various sectors [12]. Evidently, the value of data is in its openness for use and reuse. To support the use and reuse of digital resources, DARIAH EU serves as a platform to support digitally enabled research in the arts and humanities, facilitating the exchange of knowledge in digital humanities regarding content, methods, tools, and technologies. It assists researchers in utilizing these resources for building, analyzing, and interpreting digital materials while ensuring adherence to best practices, as well as to methodological and technical standards [3].

It has been suggested that the success of (digital) technology should be measured by its openness to unanticipated uses—that is, its ability to enable change (Lunenfeld 2011 as cited in [22]). The data value chain proposed by the Open Data Watch (Figure 1) points towards a crucial step between data use and reuse – the “change” step. This step emphasizes the importance of tracking tangible behavioural changes resulting from data utilisation. This framework provides a comprehensive view of the data lifecycle, from initial collection to the ultimate impact of data reuse. How to translate this into an evaluation grid for heritage institutions remains an issue to be resolved.

3.2. Managing Digital Change Responsibly: Key Concepts

In the context of the permanent social crisis we are facing, it is essential to highlight concepts that make up the framework for managing cultural heritage institutions today—namely, accounting, resilience, and sustainability are closely interrelated concepts relevant to cultural heritage management [24], particularly in today’s digitally infused reality in which we still did not fully reach digital maturity.

The Recommendation of 10.11.2021 on a common European data space for cultural heritage [12] set to maximize the opportunities created by the digital transformation and encourage Member States to help cultural heritage organisations to become more accountable, resilient and sustainable in the future. It underlines that the Member States and cultural heritage institutions should take a “holistic approach” when planning digitisation. This involves considering “the purpose of the digitisation, the target user groups, the highest quality affordable, the digital preservation of the digitised cultural heritage assets, including aspects such as formats, storage, future migrations, continuing maintenance and the necessary long-term financial and staffing resources”[12].

According to Thomas & Lamm, accountability is “an important component of ensuring pragmatic legitimacy for cultural enterprises that create value, respect the principles of sustainability (moral legitimacy), follow their mission, and deliberate and implement strategies (cognitive legitimacy)” [25]. Especially in the digital environment, accountability is an important aspect for public institutions. In other words, data is “the raw material for accountability” [26].

Resilience and sustainability are related but distinct concepts: resilience emphasizes short-term adaptability, while sustainability ensures long-term viability. Cultural resilience represents the ability of a cultural system to withstand adversity, adapt, and evolve [27]. UN Brundtland Commission has defined sustainability as “meeting the needs of the present without compromising the ability of future generations to meet their own needs” [28]. This broad definition points to heritage sector’s responsibility to ensure that heritage resources which is our inheritance from past generations remains a legacy we pass on to the future [29]. By making efficient management choices and assuming responsibility for their outcomes (being accountable), it is expected that organisations can foster both their resilience and sustainability [24].

Translating these concepts to the digital context points to the importance of the term digital maturity: “An individual’s or an organisation’s ability to use, manage, create and understand digital, in a way that is contextual (fit for their unique setting and needs), holistic (involving vision, leadership, process, culture and people) and purposeful (always aligned to the institution’s social mission)” [30]. Digital transformation is evidently a challenging process. Digitizing cultural heritage brings profound changes for heritage institutions, requiring them not only to adopt digital technologies but also to adapt to the digital landscape [31]. This involves reconsidering the representation, meaning, and, most importantly, the value of digital heritage and its impact on the community.

However, in practice it does not seem that we have reached the maturity point yet. Tanner points out that the assumptions continue to pervade that all digital things are innovative, that agile development can substitute planning, and that if competitors (or Google) are doing something, it is imperative to do it [32]. On the other hand, the case study by Marsh and others suggests that the impact concerning digitising collections can be defined by the question “What is meaningful and to whom?” [33] Thus, cultural heritage institutions need to consider their digital activities in the context of their social missions and potential value for their users’ communities and try to make sense of how digital context supports or hinders this.

3.3. From Digitalisation to Impact: Approaches and Challenges

Impact is usually described as something that brings change [33,34]. According to Tanner and Deegan [34], impact can be defined as “the measurable outcomes arising from the existence of a digital resource that demonstrate a change in the life or life opportunities of the community for which the resource is intended”. The concept of impact is rather complex. It can be tangible or intangible, positive or negative. It happens at multiple levels and magnitudes/scales. Moreover, the impact happens over time (e.g., short-term impact or long-term impact) [33]. Impact can range from individuals to communities or general society and can have various aspects, from social to economic [35].

By its very nature, the impact is difficult to assess. Most cultural heritage institutions can obtain figures on how much they have spent on their web services, how many visitors visit their site, or how the visitors navigate through their site. However, they struggle with how to relate, for example, high engagement rates to an exact measure of impact. The impact of digital cultural heritage is assessed by evaluating the value of the changes it generates. For this, it is necessary to have assessment tools or frameworks that would enable meaningful evaluations.

Various evaluation methods are used today, including impact assessment (IA). In general, evaluation processes refer to broader approaches that assess a project’s overall performance, effectiveness, and efficiency during or after implementation. In contrast, impact assessment focuses on the specific, long-term effects and outcomes of a project.

Impact assessment is defined in literature as “research that requires setting questions and choosing methods to answer them” (ISO 2014 as cited in [36]); “process of identifying the future consequences of a current or proposed action” [37]; “a tool to foster understanding of how strategic decisions about digital resources may be fostering change within our communities.” [32]; or simply “thinking before acting” (Morrison-Saunders 2018 as cited in [38]). These definitions emphasize the complex, layered process involved in impact assessment, which demands careful planning and execution. From the definitions above, we can conclude that impact involves an intervention that creates change where the effects of the intervention/project/resource are assessed in relation to its intended purpose and the potential needs of its stakeholders [39]. It reflects the difference between what would have occurred naturally and what resulted from a specific action or project.

Impact assessment has been first introduced in the heritage sector in the 1980s focusing on UNESCO’s World Heritage sites [38] but has gained more prominence in recent years. Heritage IA is closely linked to the concept of cultural capital, which includes the economic value of heritage assets [40], as well as the connection to the history of landscapes and communities, meaning the cultural values derived from heritage [41]. Additionally, cultural and social values are created through community involvement and everyday participation. Nevertheless, it’s difficult to assign a tangible sense of value to a digital resource or project.

Measuring impact supports an evidence-based approach to managing heritage. Evidence of (digital) impact can be inferred from various sources, including output data, user satisfaction measures, or performance indicators [7]. Thus, impact assessment involves both quantitative and qualitative methods and indicators. In this regard, Shaw [8] argues that a comprehensive assessment of digital collections requires a multi-faceted approach, combining statistics, surveys, user studies, usability testing, and web analytics. Yet, Tanner [32] remarks that, so far, primary measures of digital heritage have predominantly focused on web statistics, anecdotal information, or evaluations of outputs, such as the quantity of digitised materials, rather than assessing the value and change derived from these efforts.

The basic elements in measuring impact are indicators or data we interpret. Tanner proposes that indicators are the most critical part of the impact assessment because they measure progress toward set goals [32] and should reveal the change triggered by the project/action that is being evaluated. They are usually specifically developed depending on the proposed action. For example, Fukuyama and Tanner proposed a set of 13 potential indicators for the UK Web Archive (UKWA) which can be used for UKWA, but also other web archiving organisations [36]. To be usable, they must be specific, measurable, achievable, realistic, and timely (SMART) and must meet quality criteria. This means that the functionality of each indicator needs to be evaluated against established benchmarks for effective and ineffective indicators [36]. Finally, for indicators to be useful, they must be set as opposed to some baseline values that have been registered before the project has begun, and against which change can be measured. Tanner underscores that the digital domain is a challenging environment for identifying suitable indicators due to the limited availability of historical data, i.e. the absence of effective baselines, which can hinder meaningful analysis [32].

To successfully perform an impact assessment, the initial step is deciding what will be monitored and how: what data sources are relevant and available to serve as indicators, which are the relevant questions for qualitative analysis, which methods will be used [39]. All said above indicates that this is not a simple task and requires some underpinning models that would provide the framework for the IA analysis.

3.4. Exploring EU Initiatives for Impact Assessment and the Challenges of Measurement

As argued above, the adaptability of content to different types of audiences is as important for the heritage sector, as well as the increase of its engagement with digitised cultural heritage. To achieve this, we concur with Shaw’s claim that the development and preservation of high-quality digital collections that respect community standards and follow best practices is a complex and resource-intensive endeavour and engaging in their meaningful assessment further compounds this complexity [8].

The heritage sector is increasingly aware of the need to evaluate the success of its projects and the impact they have achieved on the community they serve. However, they lack knowledge and skills to fully embrace this practice. To support the heritage sector in reaching digital maturity, Culture24’s flagship collaborative action research program Let’s Get Real since 2010 provides a capacity building that enables better understanding of what impact means in the context of digital heritage. The first edition addressed the question “How to Evaluate Online Success” by exploring what success looks like for different organizations and the tools available to measure it [42]. Subsequent editions focused on various aspects of digital activities, such as aligning digital practice with social purpose, assessing whether institutions’ content is “fit for purpose”, fostering deeper human connections through digital channels, and understanding and measuring digital engagement. Such in-depth exploration helped participating organizations better understand what impact means for them, by enabling them to measure their online performance more accurately and meaningfully and thus reaching better informed decisions regarding their online activities [42].

To assess the present state of development of tools for impact assessment of (digital) heritage projects we have conducted a desk review of recent project reports that have aimed at proposing new frameworks or methods of evaluating the impact in the heritage domain. We analysed six impact assessment frameworks: the Balanced Value Impact Model (BVI), the Europeana Impact Playbook, the Change Impact Assessment Framework, Impactomatrix, the Museums of Impact Framework (MOI), and the Social Platform for Holistic Heritage Impact Model (SoPHIA); looking into their methodologies, thematic focus, and the presence or absence of proposed indicators and tried to detect challenges of measurement. We shall present our findings in the table below.

4. Discussion

Starting from the different premises, six examples described above provide various frameworks that enable heritage institutions to rethink what impact means for them in the context of their missions by providing them with tools that helps them asking relevant questions and choosing methods to answer them. While the first four examples focus exclusively on heritage in a digital context, the last two examples are addressing heritage sector in general but are still relevant for the digital heritage projects and resources.

IA frameworks such as the SoPHIA Model, MOI Framework, and Change Impact Assessment Framework primarily emphasize tangible outcomes like knowledge creation, creativity, innovation, sustainability, etc. directly addressing the scope of impact through their thematic focus. In contrast, the BVI Model and the Europeana Impact Playbook, which is based on it, focus on the fundamental values that drive these outcomes. For instance, the BVI Model utilizes five Value Lenses to guide impact assessment, ensuring a comprehensive evaluation of the core values linked to digital cultural heritage experiences. Meanwhile, Impactomatrix appears to integrate elements of both approaches.

While those IA frameworks provide different lenses on variety of impact areas and provide general guidelines for potential applications of the framework, only some of them have developed a concrete methodology guiding the assessment plan that facilitates implementing the specific approach that depends on the needs of the specific heritage institution. This highlights a key challenge: effectively translating the framework into concrete evaluation action within the unique context of each institution by deciding on evaluation goals and timeframe, research methods to be used, adequate indicators that need to be collected and interpreted, etc. This leads us to the conclusion that standards for IA concerning digital heritage are not yet agreed upon.

We can conclude that despite decades of continuous investment in the development of digital cultural resources within an ever-evolving digital landscape, there is still no clear consensus on how to assess the impact of digital heritage resources and projects. The European Commission [12] highlights the need for a “holistic approach” for Member States and cultural heritage institutions when planning digitization initiatives. This is reflected in the IA frameworks described, which take a broader view of the wider implications of such efforts. However, progress is hindered by a combination of factors: financial constraints often limit the resources available for IA, a lack of skilled personnel and institutional understanding poses a significant barrier, and the complexity of digital heritage projects makes effective assessment a challenging process.

This indicates that understanding the impact of digital resources and demonstrating the change they have produced remains a challenge. As the data value chain (Figure 1) illustrates, the ultimate goal of data use is to achieve change (through reuse), which aligns with the fundamental understanding that impact is realized through meaningful change. This principle is well-reflected in the IA models presented in Table 1. The point is that, by identifying the value of that change or by describing it as a concrete outcome, cultural heritage institutions aim to demonstrate the true significance and effectiveness of their digital efforts.

The described examples of IA frameworks developed in the recent time point to the fact that the heritage sector increasingly understand the importance of demonstrating the impact of their activities and resources on the communities they are serving. However most cultural institutions have not yet mastered the IA tools and methods to appropriate it in their work, as even with several existing IA frameworks in place, current assessments of digital heritage in majority of heritage institutions predominantly rely on metrics like web statistics and the quantity of digitized materials, overlooking the crucial need to evaluate the true value and impact of these efforts. Ultimately, at this point the key benefit of utilizing IA frameworks is the valuable learning process inherent in the evaluation itself.

To translate proposed frameworks into practical applications and strengthen the sector, achieving digital maturity and enhancing digital skills in the heritage field is essential. This requires a strategic approach that extends beyond merely sharing information on the topic. Navigating this complex landscape demands digitally literate leadership to ensure that heritage professionals acquire the necessary digital competencies and can effectively adapt to ongoing changes. Cultivating these skills across the entire team requires dedicated time and resources. In essence, cultivating a digitally literate workforce, investing in resources, and conducting evaluations thorough impact assessment frameworks are all crucial elements for building resilience in cultural heritage institutions and ensuring their sustainability.

Finally, as sustainability is becoming an integral part of EU politics, (digital) cultural heritage, impact assessment frameworks should enable lenses through which digital cultural heritage initiatives can be evaluated, not only for achieving their short-term objectives but also fostering long-term sustainability and thus contributing to the preservation and vitality of cultural heritage in the digital age. It is clear that the impact assessment of digital cultural resources is relevant for decision making processes in culture and are equally of interest for the policy makers as such assessments provide relevant data for evidence-based policy making that we are striving for.

When conducting this research, it became clear that comprehensive literature specifically addressing digital cultural heritage IA is not abundantly present. While IA frameworks are emerging in the recent years, they remain in an initial stage. Although they provide a good starting point, substantial research is still needed. Further research, including more case studies, is needed to fully evaluate the effectiveness and applicability of these IA frameworks, as a comprehensive understanding of digital initiatives requires exploring not only measurable outcomes but also the less tangible, value-driven impacts and their long-term sustainability implications.