Defining Success Criteria for Urban Living Labs: A New Urban Experience in Turkiye

1. Introduction

Urban Living Labs (ULLs) have emerged as innovative platforms for collaborative research, experimentation, and co-creation in urban environments, and they offer opportunities to develop the city with residents and other stakeholders in a real-life context [1,2]. Moreover, ULLs are characterized by their unique ability to bridge the gap between theory and practice, fostering real-world experimentation and user-centric innovation and catalyzing sustainable urban development by creating an ecosystem where ideas, technologies, and policies are tested, refined, and scaled [3,4,5]. Nevertheless, only ULLs bring together diverse stakeholders, including citizens, businesses, academia, and local governments, to address complex urban challenges and develop sustainable solutions [3,5,6,7]. The definition and the emergence of ULLs are diversified from particular perspectives. ULLs are emerging as a form of collective urban governance and experimentation to address urbanization's sustainability challenges and opportunities [1,3]. They have been used to address various urban challenges, including energy, transportation, waste management, and social inclusion [8].

Despite the growing interest in ULLs, there still needs to be more clarity on the key components contributing to their success. The success of ULLs depends on various factors, including the involvement of stakeholders, the design of the lab, the governance structure, and the political and policy context [9,10,11]. Therefore, it is necessary to identify the success criteria for ULLs to ensure their effectiveness in addressing urban challenges. However, measuring the success of ULLs presents several challenges due to their dynamic nature, diverse objectives, and multifaceted outcomes. By understanding and defining appropriate success criteria, stakeholders can effectively assess the impact and effectiveness of ULLs and improve their planning and implementation processes.

The existing literature on ULLs has extensively discussed their importance, potential benefits, and various factors contributing to their success [4,5,6,7,8,9,10,11,12,13,14,15,16,17,18]. However, there still needs to be a notable gap in the research regarding developing and validating comprehensive and standardized success criteria for ULLs. While studies have identified several factors that influence ULL success, such as multi-stakeholder partnerships, co-creation, sustainability, empowerment, transparency, and interdisciplinary engagement [4,5,6,7,8,9,10,11,12,13,14,15,16,17,18], there is a lack of consensus on the specific indicators that can be used to measure and evaluate the success of ULL initiatives.

In this regard, the article aims to identify the success criteria for ULLs by interviewing the current urban living lab cases in Turkiye. The article is organized into six subsections based on defined success criteria. However, the research provides an overview of the critical components of ULLs and their relevance for the success of ULLs and discusses the challenges and opportunities associated with ULLs and their implications for the success. As a result, the research seeks to clarify the research question " What are the measurements/criteria for a successful ULL as a new approach?” and provides insights from the perspective of the Turkish living labs. Finally, the article concludes by providing recommendations for the future development of ULLs.

2. Research Background: Evaluating the success criteria for ULLs

The literature gap lies in the need for a systematic framework or criteria to guide ULL practitioners, policymakers, and researchers in assessing ULL projects' effectiveness, impact, and sustainability. Although individual case studies and evaluations have provided insights into specific ULLs' achievements [3,6,15,18], there is a need for a more comprehensive and standardized approach to capturing and measuring success across different ULL contexts. Addressing this literature gap would involve developing a robust framework for success criteria that considers the diverse goals and objectives of ULLs, their specific thematic areas, and the unique challenges and opportunities of urban contexts. Such a framework could encompass qualitative and quantitative indicators to capture various dimensions of success, including social, economic, environmental, and governance-related outcomes. Additionally, it would be essential to establish a consensus among stakeholders on prioritizing and weighting different success criteria to ensure their relevance and applicability.

Moreover, the success of ULLs determines by various factors [14,15,16,17]. By considering success criteria, ULLs can effectively address urban challenges, global and urban problems and contribute to the sustainability of cities. The lab approaches highlight the role of fostering innovation and co-creation with external stakeholders. The situation emphasizes the significance of stakeholder engagement, network building, and trust establishment as crucial factors in the success of labs [3,6,16,18]. Although collaboration between researchers and practitioners is an essential topic, action also needs to be taken into account [19]. In this regard, the critical components and integration of ULLs as well as the importance of policy-making in the context of ULL should be underlined [9]. However, ULLs are seen as a new approach to urban planning and sustainability, particularly in experimental governance [20]. Living labs' collaborative nature and capacity for action [21] provide a strong potential for innovation, testing, and learning in urban transformation [2]. They serve as valuable tools in urban research by involving researchers, policymakers, local stakeholders, and residents in co-creation [22]. Experiments in ULLs create opportunities for social learning and empowerment and raise leadership and ownership issues in transition governance [23].

Furthermore, Yilmaz & Ertekin [5] emphasize the ULLs’ criteria as openness, continuity, realism, empowerment of users, influence, and spontaneity are not enough to measure their success [5]. The success of ULLs has to criticize for their outcomes and achievements. Various factors should be considered, including multi-stakeholder partnerships, co-creation and co-production, sustainability, empowerment, transparency, and interdisciplinary engagement. The literature highlights the importance of contextualization, politics, policymaking, and stakeholder involvement in ULLs [9,10,11]. Evaluating the success of ULLs is crucial to understand their impact and identify the key factors contributing to their effectiveness. ULLs play a crucial role in promoting sustainable urban development by providing a platform to test, refine, and scale ideas, technologies, and policies [3,4,5]. However, measuring the success of ULLs presents several challenges due to policies in which country the lab is located or multifaceted outcomes. If ULL cannot define the scale of effectiveness or the influence area, they cannot proceed to achieve good examples concerning sustainability and urban planning for our future. By understanding and defining appropriate success criteria, stakeholders can effectively assess the impact and effectiveness of ULLs and improve their planning and implementation processes.

Additionally, measuring the success of ULLs is crucial to understanding their impact on sustainable urban development. The article has highlighted vital success criteria that can be used to evaluate the effectiveness of living labs, including co-creation and collaboration [19,24,25,26,27,28,29,30,31,32,33], innovation and solution development [28,34,35], impact and effectiveness [29,36,37,38,39,40], user engagement [26,27,28,29,30,31,32,33,41,42,43,44], transferability and scalability [26,28,32,45,46,47,48,49], and knowledge creation [29,50,51,52,53]. By adopting a comprehensive and systematic approach to evaluating ULLs, stakeholders can enhance their planning, implementation, and impact assessment processes, ultimately contributing to more sustainable and livable cities.

• Co-creation and Collaboration: The active involvement and engagement of citizens, businesses, researchers, and policymakers foster a collaborative ecosystem that enhances the relevance and effectiveness of the solutions developed. Successful labs cultivate an environment where stakeholders work together, leveraging their expertise and knowledge and co-creating innovative solutions that address urban challenges [19,24,25,26,27,28,29,30,31,32,33,44]. The degree of co-creation and collaboration within a living lab can be measured by assessing stakeholder engagement, the diversity of perspectives involved, and the extent of shared decision-making. These criteria emphasize the involvement of diverse stakeholders in the innovation process and the generation of collective knowledge and solutions [19,42]. The degree of stakeholder engagement, inclusiveness of the co-creation process, and the establishment of effective collaboration networks are commonly used indicators to evaluate the success of ULLs in this dimension.
• Innovation and Solution Development: Labs serve as experimental playgrounds where new ideas, technologies, and service concepts are prototyped, tested, and refined. The success of a living lab can be evaluated based on its capacity to generate and implement innovative solutions that address specific urban challenges [21,45,55]. This criterion includes assessing the number and quality of successful innovations that emerge from the lab, their potential for scalability and transferability, and urban stakeholders' adoption of these solutions. The success criteria in this dimension focus on creating and implementing innovative products, services, or processes. Technological advancements and the ability of living labs to facilitate the adoption and integration of new technologies are essential factors to consider [21]. Indicators used to evaluate this dimension include the number and quality of innovations generated, the scalability of solutions, and their potential for market uptake and replication.
• Impact and Effectiveness: Measuring the impact and effectiveness of ULLs on urban systems is another vital success criterion. The impact and effectiveness of ULLs are critical success criteria for assessing their contribution to sustainable urban development. This dimension focuses on the tangible outcomes and benefits of living lab initiatives. Evaluating living lab projects' socio-economic, environmental, and cultural impact is crucial [40]. Indicators used in this dimension include measurable changes in urban systems or processes, improvement of quality of life for citizens, and the achievement of sustainability goals [29,36,37,38,39,40].
• User Engagement: User engagement and their satisfaction play a critical role in determining the success of ULLs. End-users, including residents and businesses, should be actively involved in the lab activities to ensure that the solutions developed meet their needs and preferences. The success criterion of user engagement encompasses factors such as user participation, satisfaction with the co-created solutions, and the perceived value of the living lab experience [26,27,28,29,30,31,40,41,42,43,44]. Evaluating user engagement and satisfaction provides insights into the user-friendliness, acceptability, and long-term viability of the solutions developed within the lab. These criteria emphasize citizens' active involvement and empowerment in the co-creation and implementation processes. User-centric approaches and incorporating end users' perspectives and feedback are essential [43,54]. Indicators used to evaluate this dimension include the level of user involvement, satisfaction surveys, and the integration of user feedback into the design and development of solutions.
• Scale: A living lab's ability to transfer knowledge, best practices, and successful solutions to other urban contexts enhances its value and sustainability. Factors such as the documentation of lessons learned, the development of implementation guidelines, and the creation of networks and partnerships for knowledge exchange contribute to the transferability and scalability of living lab outcomes. This dimension focuses on the potential for replicating and adapting innovative solutions in different urban contexts. Considering contextual factors and the transferability of knowledge and practices beyond the specific living lab setting is essential [45,49]. Indicators commonly used in this dimension include documenting and disseminating best practices, establishing partnerships for knowledge transfer, and successfully implementing solutions in other urban areas [26,28,32,45,46,47,48,49].
• Knowledge creation, policy-making and governance: Knowledge creation is intrinsic to the success of ULLs. Living labs generate valuable insights, best practices, and lessons learned through their collaborative and experimental approach. The success criterion of knowledge creation and dissemination involves assessing the quality and relevance of the knowledge produced within the living lab, its accessibility to a broader audience, and its potential to inform future urban development initiatives [29,50,51,52,53].

In addition, the impact of ULLs on policymaking processes and governance structures is also a critical success criterion [55,56,57]. By actively involving stakeholders in decision-making, living labs promote more collaborative, inclusive, and citizen-centric approaches to urban planning [22,43,56]. Evaluating their impact on policy and governance entails examining how living labs influence policy agendas, foster innovation-friendly regulations, and contribute to more effective urban governance models [58].

3. Methodological Approach

The establishment of ULLs goes beyond the mere implementation of projects and initiatives. These labs serve as platforms for capacity building, knowledge exchange, and the development of collaborative networks. They create opportunities for stakeholders to enhance their skills, share experiences, and build relationships that extend beyond the duration of specific projects. Through workshops, training programs, and collaborative activities, ULLs foster a culture of learning and collective problem-solving, empowering communities to actively participate in shaping their urban environments. This emphasis on capacity building and knowledge sharing strengthens the long-term sustainability and impact of ULLs, ensuring that the benefits and lessons learned are transferred and replicated in other contexts, ultimately contributing to a wider culture of innovation and sustainable urban development.

We aim to explore the success criteria for ULLs, shedding light on the factors contributing to their effectiveness and providing insights for future initiatives. We explore the success criteria for ULLs and their potential to promote sustainable urban development and governance by interviewing stakeholders from three (all) labs in Turkiye. The methodology consists of semi-structured interviews with key stakeholders involved in ULLs.

3.1. Semi-structured Interviews

To better understand Turkish urban living labs, we used a semi-structured interview method to collect primary data. We invited key stakeholders to participate in the interviews. Since only three urban living labs in Turkiye are registered to European Networks of Living Labs (ENoLL), we interviewed all the labs, including Başakşehir Living Labs, Bodrum Living Lab, and Mezopotamya Living Lab. We conducted in-depth interviews with representatives at the management level from each lab. The interviews consisted of 13 questions and lasted approximately one and a half hours. We conducted online interviews with all labs. All questions for the interviews were based on the section 2.

• Interview Guide: We created an interview guide to ensure consistency and promote detailed conversations. The guide consists of open-ended questions focused on topics such as the objectives and activities of the ULLs, the challenges they face, their relationships with local government, and the perceived success criteria for evaluating their impact. The semi-structured interview questions and keywords guiding to interview are given in Figure 1.

The interview questions consist of three parts. The first part contains institutional information, the position, and responsibilities of the interviewee. The second part asked questions to identify the projects, collaborations, and actors involved in the process. This section also collects information on decision-making processes, implementation processes, the project results, the advantages and disadvantages of the process, the area of influence, and the actors' roles. Finally, the third section includes questions such as the achievements, the scale of these achievements and contributions, the criteria of benefit and success, and the advantages of gathering under one roof within ENoLL.

• Limitations: It is important to acknowledge certain limitations of the research methodology. The sample size that includes three an all ULLs in Turkiye registered to ENoLL (Basaksehir Living Lab, Bodrum Living Lab, Mezopotamya Living Lab) may be relatively small, limiting the generalizability of the findings. Additionally, the article focuses on a specific context (Turkish urban living labs) and may not capture the full diversity of ULLs worldwide. We strive to incorporate diverse viewpoints in our sample selection and establish criteria that can serve as a model for all ULLs.

As a result, the methodology goal is to provide a comprehensive understanding of Turkish living labs and their success criteria. The findings will contribute to the existing body of knowledge on ULLs and serve as a valuable resource for policymakers, researchers, and practitioners seeking to enhance the effectiveness and impact of ULLs. Additionally, the research can serve as a valuable reference for policymakers, researchers, and practitioners around the globe when establishing success criteria for labs.

3.2. Data Collection and Data Analysis

The selected participants underwent individual virtual interviews. The participant's consent was obtained to audio-record the interviews in order to ensure accurate data capture. In addition to recording the interviews, detailed notes were taken for further reference. The recorded interviews were transcribed verbatim and analyzed using thematic analysis. The data were systematically coded, categorized, and organized into themes and sub-themes. This process helps identify common patterns, recurring topics, and significant insights related to the success criteria of labs.

Furthermore, the article presents the findings of interviews conducted with three prominent labs in Turkiye: Basaksehir Living Lab, Bodrum Living Lab, and Mezopotamya Living Lab. These interviews aim to gain insights into how these living labs have addressed and incorporated the identified success criteria within their initiatives and to understand the impact and outcomes they have achieved in their respective contexts.

By examining the experiences and perspectives of these ULLs, the article aims to provide a comprehensive understanding of how they have operationalized the success criteria within their specific contexts. The interviews reveal valuable insights and lessons learned that could inform the design and implementation of future lab initiatives. Furthermore, the findings contribute to the existing knowledge base on ULLs, shedding light on their potential to address urban challenges, foster collaboration, and drive sustainable urban development.

In addition to the insights gained from the interviews, it is essential to note that ULLs offer unique opportunities for experimentation, learning, and adaptation. These labs provide a real-life testbed where stakeholders can pilot innovative ideas, assess feasibility, and make necessary adjustments based on feedback and evaluation. This iterative process fosters a culture of continuous improvement and empowers cities to respond effectively to evolving urban challenges. By embracing a flexible and adaptive approach, ULLs have the potential to serve as catalysts for transformative change, driving the transition towards more sustainable and resilient urban futures. The integration of success criteria within the operations of ULLs helps ensure that these transformative efforts are effectively measured, evaluated, and shared, contributing to advancing urban innovation and sustainable development on a broader scale.

4. Major Findings and Comparative Analysis of Turkish Living Labs

The subsequent sections delve into the essential findings and analysis from the interviews with Basaksehir Living Lab, Bodrum Living Lab, and Mezopotamya Living Lab. The in-depth exploration provides a comprehensive understanding of their approaches, outcomes, and the extent to which they have successfully integrated the identified success criteria.

4.1. Instutional Structure and Budget Resources

Basaksehir Living Lab, located in Istanbul, is a unique collaboration platform between stakeholders, including the municipality, residents, businesses, and academic institutions. Basaksehir Living Lab is a unit under the Innovation and Technology Directorate of Basaksehir Municipality (Istanbul) without a legal personality. Every year, through an open tender method, additional personnel are determined to supplement the municipality's staff, and an annual work plan is established. While activities prioritizing the creation of awareness about new technologies form the focus, organizing educational activities to enhance technology literacy in the city and developing future technologies such as drones, virtual reality, and e-sports constitute the core of their work. Basaksehir Municipality covers the expenses for events, implementation projects, and employees. In addition, sponsorship revenues from events, European Union project applications, and international funds contribute to the lab's budget.

On the other hand, Bodrum Living Lab, situated in the coastal town of Bodrum, focuses on sustainability and the preservation of the natural and cultural heritage of the region. Bodrum Living Lab is established as a private company in Bodrum and collaborates with local urban actors. Bodrum Living Lab prioritizes urban development studies while also incorporating educational activities. Applying information technologies to social projects is one of their main goals. Additionally, agriculture, tourism, health, and urban planning are the primary focus areas of Bodrum Living Lab. Their approach is to adapt their work from a macro to micro scale, with rural development as the central axis. Furthermore, the lab's budget (including employee expenses, events, and applications) is provided by the private sector (the founding company). They are also currently engaged in the process of applying for European Union projects to secure additional funding.

Mezopotamya Living Lab, located in Sanliurfa- eastern of Turkiye, is an initiative that aims to create a sustainable living environment and promote innovative solutions. Mezopotamya Living Labs’ one of the primary goal is to develop and implement sustainable practices related to water management, agriculture, energy, and waste management. Education and capacity building are crucial components of the Mezopotamya Living Lab. Workshops, training programs, and awareness campaigns are conducted to empower local communities and stakeholders with the knowledge and skills necessary to implement sustainable practices. The collaborative approach that lab embark on encourages the sharing of knowledge, expertise, and resources to develop practical solutions for sustainable development and smart cities. Furthermore, the lab's budget (including employee expenses, events, and applications) is provided by Sanliurfa Municipality Department of Smart Cities, Strategy and Innovation. The lab is also currently engaged in the process of applying for European Union projects to secure additional funding.

However, Table 1 presents an overview of three ULLs in the Turkish context, highlighting their focus, mission, partnership, and leadership. Main Focus indicates the primary focus or area of interest for each ULL. The listed ULLs have different focuses, such as methodology and research, problem-oriented approach, education-oriented initiatives, and user-oriented. This aspect defines the core objective or thematic area that guides the activities and projects within each ULL. The mission describes the overall purpose or goal of each ULL. It provides a brief statement about the specific mission of the ULL, which typically reflects the desired outcomes, impact, or benefits the ULL aims to achieve. The missions mentioned in the table include increasing technological awareness, expanding the implementation field of information technologies, and generating data for urban planning. However, the partnership outlines the key partners involved in each ULL. Partnerships play a crucial role in ULLs as they bring together various stakeholders from different sectors to collaborate and contribute to the initiatives. The listed partners typically include government institutions, universities, and non-governmental organizations (NGOs). The specific composition of the partnership may vary depending on the ULL and its objectives. The leadership indicates the entity or organization responsible for leading and coordinating the ULL. In this context, the mentioned leadership entities include government agencies and private companies. The designated leadership takes on the responsibility of guiding the ULL activities, facilitating collaboration among partners, and ensuring the fulfillment of the ULL's objectives. In the end, Table 1 provides a concise overview of the main features and characteristics of the selected ULLs in the Turkish context. It helps to understand the specific focus, mission, partnership composition, and leadership structure of each ULL, highlighting their unique attributes within the broader ULL landscape.

4.2. Innovation and Solution Development

"The region's needs should be analyzed, and a database should be established; infrastructure development should be prioritized.".

Bodrum Living Lab, May 18

Within the scope of Başakşehir Living Lab projects, it enables the production and testing of new technologies and carries out implementation projects within the boundaries of Başakşehir Municipality. Additionally, the lab participates in projects within the European Union, providing pilot areas for implementation activities through partnerships. By applying the newly developed technological approaches produced within the lab to the city at the building or neighborhood scale, it seeks to address local problems while also displaying approaches that generate solutions on a global scale in line with the SDGs. Furthermore, with priority given to spatial implementation through pilot projects, infrastructure is being prepared to serve as an example for the district and city where the lab is located and other cities and countries.

On the other hand, Bodrum Living Lab focuses on conducting activities that prioritize social entrepreneurship. In collaboration with local government, academia, and local actors, it carries out spatial studies and applications with urban planning as a priority for the city of Bodrum. Through projects implemented within the lab, it contributes to increasing knowledge and fostering digital dialogue among the citizens in the region. It is worth highlighting that applying information technology to social projects in Bodrum Living Lab focuses on different problems and issues compared to other labs in Turkiye.

However, Mezopotamya Living Labs’ aim to enable the production and testing of new technologies and carries out implementation projects within the boundaries of Sanliurfa Municipality. Besides, lab focuses on educational activities for residents to develop understanding and practicing new technologies in the field of planning and design.

4.3. Impact, Effectiveness, User Engagement and Satisfaction

In the context of ULLs, the term "user" refers to individuals or groups who actively participate in the co-creation, experimentation, and testing of innovative solutions within the urban environment. Users are not passive recipients but rather active contributors in developing and implementing urban interventions. Users in ULLs can include a wide range of stakeholders such as residents, citizens, community organizations, businesses, researchers, government agencies, and non-governmental organizations (NGOs). They bring diverse perspectives, expertise, and experiences to the collaborative process, ensuring that the solutions developed are relevant, effective, and responsive to the needs and aspirations of the urban community. Their firsthand experiences and local knowledge contribute to identifying problems and designing appropriate solutions. Through their involvement, they provide feedback, evaluate prototypes, and help refine and improve the proposed solutions. This iterative process allows for the development of context-specific and user-centric interventions. By involving users from the beginning, ULLs create a sense of ownership and empowerment among the stakeholders, increasing the likelihood of successful implementation and long-term sustainability. User’s role, user’s size and involvement levels can measure the impact and effectiveness.

However, the measurement and evaluation of the impact of lab activities on urban, regional, national, and international scales are vital factors that guide and assess the success of lab work. According to the results of interviews conducted with labs in Turkiye, including Basaksehir Living Lab, Bodrum Living Lab, and Mezopotamya Living Lab, they are actively working on developing an impact assessment system through their academic collaborations, despite having yet to establish an established system.

Additionally, Basaksehir Living Lab conducts surveys and field studies (with a predetermined sample size) to measure satisfaction with the lab's activities and events. Furthermore, they highlight that the transition of some projects within the entrepreneurial ecosystem they have built into companies and the stage of international product development also demonstrate the reach of the lab's impact. With the increase in online platforms for events and training during the pandemic, there has been an observed increase in the number of users in Basaksehir Living Lab. This situation has also brought about diversity among the users. As a result, users can not only participate in activities and training but also act as actors in planning and design projects.

Besides, expanding the entrepreneurial ecosystem through the mentorship and spatial opportunities provided by the lab has increased participation in lab activities. Recently, the lab has begun to expand its activities through spatial implementation projects and testing in pilot areas. Although the success evaluation criteria are not yet clearly defined at this stage, the increase in the number of users in Basaksehir Living Lab, the expansion of entrepreneurial activities, and satisfaction with the implementation projects indicate that the lab is taking the proper steps.

On the other hand, Bodrum Living Lab and Mezopotamya Living Lab are a recently established laboratories that have just begun their work and have yet to reach the stage of impact measurement. However, from the interview results, it is clear that they have observed positive user and collaboration ecosystem changes because they primarily focus on finding local responses to urban problems.

4.4. Scale

From its initial years of the establishment until 2019, the Basaksehir Living Lab focused on providing an environment and conducting education-oriented activities, while between 2019 and 2023 witnessed research and user-oriented tasks. On the other hand, Bodrum Living Lab conducted problem-oriented work, while Mezopotamya Living Lab focuses on the educational activities. Although the three lab models have different priority areas, content, and ecosystem networks, their projects are fundamentally rooted in the urban scale. The distribution of users in the projects conducted within these labs is not publicly disclosed on open platforms. The topics shared online by the three labs include project content, collaborations, and project results. This situation raises the question of whether the labs concede with the criteria of openness, empowerment of users, and transferability. The interviews conducted reveal certain limitations regarding the topic. For example, the size/scale of the users varies depending on the activity or project. The preferences of the users also influence the scale of participation. However, collaborations also vary depending on the projects and activities. In fact, the intended project impact is related to the scale. For instance, the waste project (smart waste-pilot implementation) conducted by Basaksehir Living Lab has served as an example for other municipalities and contributed to the dissemination of this project.

Still, in the end, there needs to be more transparent information about the topic for all labs. The limitation here could be the planning system in Turkiye. However, since local governments and ULLs are not fully cooperating (except for the institution to which lab is affiliated) and there needs to be more information about labs in Turkiye, it enhances complicated to interpret.

4.5. Knowledge creation

“There are successful and implemented projects/products in the entrepreneurial ecosystem that have achieved commercial success. This success in the lab increases the number of entrepreneurship applications.”.

Basaksehir Living Lab, May 25

The European Network of Living Labs (ENoLL) offers services to member laboratories to benefit from the lab ecosystem, strengthen collaborations, and support knowledge-based development. Coming together under a single umbrella provides advantages for labs, such as strengthening collaborations, mentorship services and staying informed about new developments. In recent years, ENoLL has focused on transforming labs from being centered on real users and real applications to becoming project hubs. Previously, approximately 80% of the ENoLL budget came from memberships and around 20% from projects, but this ratio has reversed due to changes in the LL concept explained by the European Commission. Project revenues account for approximately 80% of the ENoLL budget, indicating increased importance placed on labs. The average annual budget has also increased from around 400,000 euros to 1,5 million euros, demonstrating the growing significance of labs. The increased emphasis on labs in urban planning signifies the beginning of a new era. Achieving the SDGs is a current agenda object within this new approach. Labs' ability to expedite the implementation process for local governments, test and implement innovation-driven initiatives, and foster collaboration with diverse urban actors is considered advantageous in the new planning era.

Additionally, labs focusing on local problems have the potential to address global issues as well. For example, Basaksehir Living Lab's energy, recycling, and waste-focused projects aim to address Turkiye's environmental issues while contributing innovations to the global agenda. The dissemination of knowledge becomes crucial in this context. While Basaksehir LL, Mezopotamya LL and Bodrum LL, as ENoLL members, can share their knowledge on international platforms. However, in their search to address local problems, all three labs have opened the path for knowledge production and sharing in Turkiye through their activities, projects, and project outcomes.

4.6. Co-creation, Collaboration, Policy-making and Governance

“The construction of the science center building is ongoing. However, during this time, we continue our face-to-face education programs. Our education programs are open to everyone without any criteria, and individuals between the ages of 7 and 40 can participate in the training sessions free of charge. The only disadvantage we have in this regard is that participants from outside Urfa do not attend the training. Trainings are provided on various topics such as mobile application development, coding, entrepreneurship, and software development.”.

Mezopotamya Living Lab, June 13

Collaborations, project-based participant numbers, and roles vary within the scope of lab studies. The labs interviewed in Turkiye have a broad collaboration network, and their activities and projects attract participants. However, this does not necessarily indicate a specific model of participation and collaboration. According to the interviews conducted, although there is an increase in interest in lab studies, the extent of their impact cannot be measured. Not evaluating the impact area also affects the measurement of lab success criteria. To efficiently support urban problems through labs, it is advantageous for each lab to have its model. For example, defining scale in terms of projects, impacts, participation, and collaboration; describing the ecosystem approach in terms of collaboration level, education level, participation level, implementation level, infrastructure level, and entrepreneurship level; and addressing actor diversity with user roles, designer roles, and lifespan in experimental governance processes (such as decision-making, collaboration, implementation, evaluation and co-creation process, and process of re-thinking and re-acting) can provide perspectives for benefiting from labs at each level. Thus, selecting methods and determining influential factors and success criteria can create a new participation structure. In this way, we will benefit from the reflections of the new era, which we define as solid cooperation and entrepreneurship.

"The decision-making process in Turkiye is very slow, and reaching a conclusive decision is very difficult. With ULL, local relationships have improved and work patterns have been disrupted. The LL should govern the regulation and implementation process.".

Bodrum Living Lab, May 18

The necessity of developing/implementing innovative products and services by labs to add value to stakeholders has also been discussed. For this, it is proposed to initiate efforts to demonstrate the add value of LL and make it a national policy. Currently, labs in Turkiye are responsible for coordinating projects and activities on behalf of cities while local governments carry out the implementation. Labs are positioned as guiding entities. This situation can prevent the proliferation of labs. Therefore, considering the lab model within the framework of experimental governance and incorporating it into countries' planning systems is seen as a step for the future.

Additionally, during the interviews, it emerged that labs in Turkiye need support regarding funding mechanisms. In other words, it was emphasized that labs require additional financial support. However, the budget issue is also related to the management and planning system. Although different institutions provide project support, these supports are primarily focused on academia, and there is no specific funding mechanism for labs in Turkiye. The possibility of support from institutions such as the Istanbul Development Agency was also discussed during the interview. Establishing labs under local governments, the private sector, or academia may also have an impact. It is necessary to give legal personality to labs in Turkiye and define them within a legal framework, which is essential for becoming centers of attraction and entrepreneurial support.

5. Discussion

ULLs have gained significant attention in recent years as innovative platforms [7,17,20,23,36]. In order to evaluate the effectiveness of ULLs, it is crucial to define and understand the success criteria that guide their operation and impact. For this purpose, the article aims to shed light on the success criteria for ULLs for innovation and sustainable development. By interviewing ULLs in Turkiye, examining the insights and findings from the literature, we explored the key dimensions, challenges, and potential benefits associated with ULLs. Through a multidisciplinary approach, ULLs offer a promising avenue to address the complex urban challenges of the 21st century and foster collaboration among diverse stakeholders in creating more inclusive, sustainable, and resilient cities [21,22,37]. By examining Başakşehir Living Lab, Bodrum Living Lab, and Mezopotamya Living Lab, we have identified critical success criteria that contribute to the effectiveness and impact of these ULLs and it can be an example for worldwide labs.

In fact, ULL is active stakeholder engagement. All three labs in Turkiye have recognized the significance of involving diverse stakeholders, including local government, academia, businesses, and citizens. These labs have captured valuable insights, local knowledge, and different perspectives through workshops, educational activities, competitions, and collaborative platforms. This inclusive approach ensures that the solutions and initiatives developed are relevant, practical, and meet the actual needs of the communities they serve.

Additionally, collaboration emerges as critical success criterion for ULLs. Başakşehir Living Lab, Bodrum Living Lab, and Mezopotamya Living Lab have successfully fostered partnerships and collaborations with various urban actors. These collaborations facilitate knowledge exchange, resource sharing, and joint problem-solving efforts. By bringing together different stakeholders, including researchers, policymakers, and industry representatives, these labs create a collaborative ecosystem that drives innovation and creates a multiplier effect regarding their impact. On the other hand, by ensuring that the voices and perspectives of all community members are taken into account, labs can promote social equity and inclusivity in their initiatives. For instance, Başakşehir Living Lab, Bodrum Living Lab, and Mezopotamya Living Lab have demonstrated their commitment to inclusivity by engaging with citizens, marginalized groups, and other often-underrepresented stakeholders. This approach fosters a sense of ownership and empowerment within the community, leading to more sustainable and inclusive urban development outcomes. In addition, these labs can affect the other regions apart from where lab located. Therefore, scale for collaboration is crucial measurement element.

The dynamic nature of urban challenges requires labs to be agile and responsive in their approaches. Başakşehir Living Lab, Bodrum Living Lab, and Mezopotamya Living Lab have shown flexibility and adaptability by continuously revisiting their strategies, methodologies, and projects to address emerging needs and changing circumstances. This adaptability allows living labs to stay relevant, innovative, and effective in tackling complex urban issues. On the other hand, securing and efficiently allocating resources is crucial for implementing projects and initiatives. Except Mezopotamya Living Lab, Başakşehir Living Lab and Bodrum Living Lab have leveraged diverse funding sources; including government support, sponsorships, and European Union project funding successfully. This resource management approach ensures the sustainability and continuity of their activities, enabling them to influence their communities long-term. However, Başakşehir Living Lab, Bodrum Living Lab, and Mezopotamya Living Lab exemplify successful Turkish labs that have demonstrated their commitment to stakeholder engagement, collaboration, inclusivity, adaptability, and resource management. By adopting these success criteria, ULLs can effectively address urban challenges, drive innovation, and foster sustainable development in Turkiye and worldwide. As urbanization and the need for sustainable urban solutions continue to grow, the success criteria identified in the article provide valuable guidance for establishing and operating future ULLs in Turkiye and beyond. By embracing these criteria, living labs can enhance their impact, empower communities, and contribute to building more resilient, inclusive, and sustainable cities.

Table 2 illustrates a four-stage evaluation scale for lab success criteria. From the interviews, it is understood that the first level is education for co-creation, while the highest level is indicated as product production and entrepreneurship. Furthermore, the criteria of collaboration and, innovation & solution development can be measured at local, urban, national, and international levels. However, the labs in Turkish case try to manage their success with impact area and effectiveness factor. They do not achieve the results and expectations yet but the labs focus on the getting feedbacks from the users. Therefore, short surveys, observations or ethnography studies have been undertaken by the labs in Turkiye to address success of the lab projects. Additionally, for the knowledge creation criterion, the first level is brief updates, while the final level is sharing all implementation activities, project details, and outcomes are taken into account by the labs.

Through stakeholder interviews, these living labs have identified critical factors for success and effectively implemented their projects. Although the concept of labs in the Turkish context has not spread as widely as in Europe, labs have started to be established in different regions in recent years. While they have yet to be fully integrated into the country's planning system, the implementation efforts carried out in collaboration with local governments are promising in urban planning. However, labs have achieved success in many areas in order to respond to the global agenda and generate solutions to local problems. In this context, the fourth section, which includes the results of in-depth interviews, evaluates the three labs comparatively.

The three labs considered within the scope of determining the success criteria are presented in Figure 2. Basaksehir Living Lab stands out as advanced and successful labs in terms of the success criteria defined in the article. Additionally, the broad scope of influence demonstrates the lab projects' success. However, expanding urban planning and design studies in the labs and educational activities contributes to increasing the level of impact in the field, offering a different perspective. Although Bodrum Living Lab and Mezopotamya Living Lab are relatively just established labs, they have shown success in various areas. While the level of impact and scale has yet to be determined, their progress toward success can be understood from the conducted studies.

However, the success of Başakşehir Living Lab, Bodrum Living Lab, and Mezopotamya Living Lab cannot be measurable from any research except the interview, since there is no open information about the progress on their websites. By engaging with stakeholders, these labs have established strong partnerships, fostered collaboration, and effectively addressed local and global challenges. In addition, the interview has facilitated a participatory approach, ensuring that the voices and perspectives of various stakeholders are considered in the decision-making process.

Furthermore, Turkish ULLs have shown their dedication to engaging with stakeholders and collaborating, promoting inclusivity, adaptability, and efficiently managing resources. Although there have been notable achievements and positive outcomes with the ULLs in Turkiye, it is essential to recognize that challenges and limitations must be addressed. Factors such as varying participant sizes, preferences, and collaboration dynamics can influence the effectiveness of ULLs. Future research should enhance the openness, user empowerment, and transferability criteria of ULLs to ensure their continued growth and effectiveness in addressing complex urban challenges, fostering innovation, and creating sustainable cities for the future. By understanding and implementing the identified success criteria, ULLs can unlock their full potential and significantly contribute to sustainable urban development.

6. Conclusions

The article provides an overview of the key components of ULLs and their relevance for the success of ULLs by discussing the challenges and opportunities. Through interviews conducted with prominent ULLs, such as Basaksehir Living Lab, Bodrum Living Lab, and Mezopotamya Living Lab, valuable insights have been gained regarding the implementation of these success criteria. Each lab highlighted distinct characteristics and approaches, highlighting the diversity and richness of the urban living lab ecosystem. We explored the concept of ULLs as transformative platforms for innovation and sustainable development, specifically within the context of Turkiye. The success criteria identified in the research encompass various dimensions, including innovation and solution development, impact and effectiveness, user engagement, scale, knowledge creation, co-creation, collaboration, policy-making, governance, and resource management.

While there are notable achievements and positive outcomes associated with these labs, challenges and limitations also exist. Varying participant sizes, preferences, and collaboration dynamics can influence the effectiveness of the labs. Further examination is necessary to enhance the openness, user empowerment, and transferability criteria of ULLs. Continued research and practice in this field will further enhance the effectiveness of ULLs and contribute to their growth and adoption worldwide. Ultimately, the leading goal is to create more inclusive, sustainable, and resilient cities that meet the evolving needs of urban populations. Understanding how ULLs contribute to sustained urban transformation, policy changes, and achieving broader sustainable development goals would provide valuable insights into their success. By addressing this literature gap and developing a comprehensive framework for success criteria, researchers and practitioners can enhance the evaluation and monitoring of ULLs, enabling evidence-based decision-making, fostering learning, and facilitating knowledge exchange within the ULL community.