Evaluation of Public ICT-based Real Estate Development Projects and Analysis of Operational Performance of Related Companies: Focusing on Smart City Projects in Korea

Jaehwan Kim; Heecheol Shim

doi:10.20944/preprints202308.1879.v1

Submitted:

25 August 2023

Posted:

29 August 2023

You are already at the latest version

Abstract

This study identified, analysed, and prioritised significant factors for standardising the Korean smart city project evaluation system. We analysed the efficiency and productivity of companies currently providing smart city services to consider both policy and practical aspects. The prioritisation of smart city planning reflects the latest trends in South Korea, where urban planning is moving towards smart city planning. Furthermore, the capacity building of public-private/private-private partnerships indicate the importance of business scalability. This indicates that smart city services are only stabilised when the private sector is involved and leads the project, rather than focusing on public development. The feasibility of building intelligent facilities indicates that smart city projects should be implemented after securing cost-benefit feasibility. The results were used as the basis for building an evaluation system, showing that in smart city convergence alliances, small- and medium-sized enterprises achieved the highest efficiency by reducing inputs to 81% and 86%, under the assumptions of constant and variable returns to scales, respectively. As the operational aspect is more problematic than the technical aspect, policy alternatives are necessary for smooth business progress, including increased flexibility of laws and institutions and the activation of policies that temporarily relieve regulations to demonstrate smart city projects.

Keywords:

smart city

;

evaluation system

;

smart city convergence alliance enterprises

;

data envelopment analysis

;

operational efficiency

Subject:

Social Sciences - Urban Studies and Planning

1. Introduction

A smart city is based on intelligent infrastructure in which technology is applied to solve existing urban problems and improve citizens’ quality of life [1]. Recently, smart cities are not only being viewed as objects; their non-physical elements are also being emphasised [2], building a foundation to address social problems in collaboration with experts and citizens [3]. The importance and necessity for smart cities is rising because, owing to the COVID-19 pandemic, activities that previously took place in physical spaces have shifted to virtual ones [4]. In recent years, smart cities have been ranked high on the agenda of urban discourse because of significant changes in daily life, including rapid urbanisation rates, a decline of old downtowns, and socio-economic and environmental factors [5]. According to Mckinsey’s market research report, the smart city market is expected to grow up to $1.7 trillion by 2025, and this trend will continue to expand, as most of the countries worldwide, including developing ones, are promoting smart city projects [6]. Although a smart city project is highly important as a national policy, and demonstration projects are actively underway, they are in a relatively early stage, and the actual field application and current-status analysis are often insufficient. Therefore, many are demanding the reorganisation of the public project evaluation system [7]. Given the nature of these projects, which require large amounts of capital, it is necessary to improve their return on investment (ROI); however, no universal performance management system or evaluation indicators are internationally accepted [8]. Deriving standardised evaluation indicators is difficult despite their importance because of the vastness of the concept and comprehensiveness of the project, as well as the lack of adequate performance management systems of companies that actually carry out the project, based on which evaluation factors should be derived [9]. A smart city is a fusion and convergence concept that goes beyond simple urban planning; encompasses technology, economy, society, and politics; and requires cooperation between stakeholders and an organic relationship with the city government. We have been witnessing side effects, including the implementation of multiple simultaneous projects by several government departments and a business model that focuses on building infrastructure to achieve short-term results [10]. Consequently, evaluation indicators vary based on department, project, and participating organisations, and they are not standardised. South Korea operates a smart city convergence alliance, comprising enterprises, universities, research institutes, and associations; however, their operational efficiency is not measured, and they fail to set the future direction and improve the sustainability of the business [11]. In particular, small- and medium-sized enterprises (SMEs) are vulnerable to management risk, based on the impact of the business, and monitoring and consulting on performance is essential. Against this backdrop, this study established an analysis model and derived implications and improvement suggestions for the smart city project evaluation system. First, by reviewing the evaluation system of smart city projects that have been promoted in various forms and collecting opinions through expert focus group interviews (FGIs), we set standard indicators to measure the importance of each evaluation factor and set priorities. Second, we conducted an efficiency and productivity analysis on 38 SMEs, which are most sensitive to business performance among the currently operating participants of the smart city convergence alliance and thus use them as the basis for an evaluation system along with priorities set by evaluation factors. Third, based on the analysis results, we suggested improvements and implications for a future evaluation system and management performance in the operational stage.

1.1. Methodology

This study reviewed the literature on smart city project evaluation systems and reclassified items based on the criteria used in the evaluation. First, we organised the overlapping factors and established the first tiers as technical expertise, plan specificity, sustainability, and scalability. Technical expertise was measured by technology acceptance and administrative accessibility, and plan specificity was divided into economic, urban, and educational infrastructures, with a focus on economic infrastructure. Sustainability, which focuses on social infrastructure, was divided into social, living, and environmental infrastructures, whereas scalability was divided into the cultivation of specialised human resources and the formation of an industrial ecosystem, to derive the second tier. The third tier was selected through five rounds of FGIs to compare the importance of the various types of factors. To standardise the evaluation indicators, they were established as generic indicators and had various implications. The details of the three tiers are discussed in the section on the derivation of the hierarchy diagram. Based on the final hierarchy diagram, we derived priorities based on the item, which were expected to serve as base materials for the establishment of an evaluation system. Next, we measured SMEs’ efficiency and productivity; which are most sensitive to business performance among the participants of the current smart city convergence alliance. Based on the results of the analysis, we provide suggestions and implications to improve a practical evaluation system that combines policy and company operability.

1.2. Scope of the Research

First, a survey was conducted to derive the analytic hierarchy process (AHP) analysis results using the first analysis methodology, that is, the hierarchy diagram. A questionnaire was distributed to 100 experts working on smart cities. A total of 100 questionnaires were collected and 83 valid samples were used, excluding inconsistent responses (17). Survey participants were evenly distributed: 23 university professors, 20 public officials, 17 general workers, 12 researchers, and 11 employees at public corporations. Next, in addition to the qualitative results gathered in the previous analysis, quantitative, efficiency, and productivity analyses were performed to overcome the limitations of the study. The operational status of smart city convergence alliance companies currently in operation was examined. The analysis point was limited to three years from 2019 to 2021, and an analysis model was established to identify changes in trends over the three years. Six variables were used in the analysis, details of which are presented in the results. Ultimately, these results complement the limitations of prior studies by employing both qualitative and quantitative perspectives, which present diverse implications, as the viewpoints of each analysis method coincide.

Table 1. Analysis model design.

Identification of priority items	- Conducted an expert survey based on a finalised hierarchy diagram- Utilised AHP analysis to determine the weight of each item- Identified priorities based on the derived weights
Efficiency and productivity analytics	- Measured efficiency and productivity of 38 SMEs in a smart city convergence alliance - Conducted year-by-year analysis based on data from 2019 to 2021- Identified the flow of change through trend analysis rather than a simple point-in-time basis analysis
Analysis results and conclusions	- Provided improvements and implications for the future evaluation systems based on the priorities, efficiency, and productivity analysis results

2. Theoretical Considerations

2.1. Smart City Overview

Previous studies on information and cyber cities tended to introduce the smart city concept as the ‘Wired City’ [12] or ‘Digital City’ [13,14]. However, this characterisation has been criticised for being overly technology-driven, focusing on corporate needs, and failing to accommodate urban needs and incorporating social and environmental sustainability [15,16]. In response to this criticism, the Organization for Economic Cooperation and Development (OECD) [17] defines smart cities as ‘initiatives or approaches that effectively leverage digitalisation to boost citizen well-being and deliver more efficient, sustainable, and inclusive urban services and environments as part of a collaborative, multi-stakeholder process’. The International Telecommunication Union (ITU-T) [18] defines a smart city as ‘an innovative city that uses information and communication technologies (ICTs) and other means to improve the quality of life, efficiency of urban operation and services, and competitiveness, while ensuring it meets the needs of present and future generations with respect to economic, social, and environmental aspects’. In the promotion of smart cities, solving urban problems based on digitalisation and ICT has been discussed along with the importance of governance owing to the convergence of ICTs and digitalisation. Consequently, smart cities are evolving to prioritise citizens rather than technology and embrace inclusion, sustainability, and resilience. According to Cohen [19], smart cities will evolve into spaces that incorporate citizens when comprehensively demonstrating various technologies and services; although the introduction and operation of related facilities is essential for providing services using digital infrastructure, it is not sufficient for discovering and providing the services that citizens need. During this process, innovation emerges and spreads, promoting the growth of related industries.

Therefore, the public and private sectors must work together to demonstrate and spread smart cities. A digital infrastructure is needed, and services must be provided based on it. However, even with digital infrastructure, services will not be provided automatically. Smart city policies are being promoted, recognising that various factors such as digital infrastructure and technology, public-private governance system construction, and institutional support are required to provide smart city services [20,21]. In this context, smart city policies in Europe have implications for the spread and distribution of smart cities in South Korea. As a leader in emerging urban issues such as green cities and eco-cities, Europe is also taking the initiative in smart city policies [22]. The European Union’s smart city policy is driven by a program called the European Innovation Partnership on Smart Cities and Communities (EIP-SCC) [23], which comprehensively promotes the demonstration and spread of smart cities, performance evaluation and verification of projects, standardisation, and establishment of a knowledge platform, providing significant implications for smart city promotion policies [24].

2.2. Discussion on Evaluation Indicators for Smart City Projects in South Korea

Smart city projects in Korea are categorised into Smart Challenge, Smart Town, Core Technology Discovery, and Smart Campus Challenge Projects. In this section, the evaluation indicators of each project are reviewed to establish a hierarchy diagram for the AHP analysis. The evaluation indicators for each project can be listed as follows. First, the Smart Challenge Project aims to solve urban problems by utilising creative ideas (solutions) from the private sector, including companies and universities, and spread the best solutions to other local governments and overseas. Unlike past local government support programs, companies, universities (ideas and investments), and local governments (space provision and citizen engagement) were involved in the project from the beginning. Based on these characteristics, smart city project evaluation indicators comprised five written assessment items and two presentation items [25].

Table 2. 2021 Smart Challenge Project evaluation indicators.

Div.	Item			Score	Sub-items
Written assessment (60%)	Necessity and feasibility of project			20	- Analysis of the current situation and problems at the target site and feasibility of solutions - Goals and objectives and the need to implement preliminary and main projects - Specificity and achievability of quantitative performance goals (KPIs) for problem-solving - Whether the local government has established (or is establishing) a smart city plan
	Excellence and innovation of the preliminary project * Demonstration plan of key solutions			30	- Appropriateness and innovativeness of key solutions to achieve goals - Feasibility of demonstration and adoption of the proposed solution - Economic and social effects of the proposed solution - Appropriateness of securing, linking, integrating, and utilising data collected by the solution - Domestic and international spread of the proposed solution (commercialisation, business model, etc.) - Post-operation and management plan for solutions and services
	Promotion system adequacy	Collaboration of citizens & stakeholders		10	- Diversity of tools for citizen engagement and stakeholder conflict control - Specificity of citizen engagement and stakeholder conflict control measures based on project phase
		Interagency collaboration		20	- Uniqueness and excellence of the governance system for preliminary project - Participation and recruitment plan of private companies - Financing and investment plans of local governments, companies, etc.
	Specificity and excellence of the main plan			20	- Basic direction, promotion strategy, and implementation plan of the main project (phased roadmap, etc.) - Specificity of sustainable promotion measures such as management and operation - Specificity of the plan to secure the budget for the main project
Presentation (40%)	Commitment to the project implementation		Local government commitment and feasibility	35	- Excellence of local government conditions and capacity to promote smart cities - Appropriateness of project goals and specificity of local government support and strategies - Feasibility of the local government’s budget and investment plan - Measures to ensure the sustainability of solution operations and management and service provisions
			Private actors’ willingness to commercialise	35	- Appropriateness of plans to engage, recruit, and cooperate with private companies, etc. - Feasibility of the financing and investment plans of companies in implementing the project - Domestic and international expansion plans, including commercialisation of the solutions - Measures to promote sustainable smart cities, including the introduction of business models
	Expected effects and sustainability of the project			30	- Innovation, excellence, and sustainability of the proposed key solutions - Potential to comprehensively solve urban problems through preliminary and main projects - Feasibility of the project management plan for visible results - Measures to achieve socio-economic ripple effects such as enhancing regional competitiveness and job creation

Second, the Smart Town Challenge Project was promoted to create region-specific smart towns within medium- and small-sized cities to lay the foundation for the spread of smart solutions to other local governments through demonstrations, while considering local needs and conditions. Based on these characteristics, the evaluation indicators for the Smart Town Project comprised four written assessment items and two presentation items [26].

Table 3. 2021 Smart Town Project evaluation indicators.

Div.	Item	Score	Sub-items
Written assessment (60%)	Necessity and feasibility of the project	20	- Validity of target sites’ status and need analysis - Specificity and feasibility of vision, goals, and strategies - Specificity and feasibility of solution-specific KPIs
	Excellence of the project plan	30	- Alignment with the project type (theme) and alignment between solutions - Track record of identifying local demand for the project plan - Excellence and specificity of the project plan
	Adequacy of the project plan	30	- Specificity and excellence of the project implementation organisation and cooperation governance structure and operation plan - Performance of preliminary consultations with related organisations and departments for construction, operation, and management, as well as specificity of future cooperation plans - Excellence of project management plan to prevent delays in project implementation - Specificity of local government financing plan
	Sustainability of the project	20	- Specificity of the post-operational management plan - Possibility of spreading to other local governments and expected effects
Presentation (40%)	Commitment to the project	50	- Commitment to the project and specificity of the implementation plan - Does the business plan reflect local conditions? - Excellence and specificity of the business plan - Feasibility of the local government’s business plan, including securing budget and cooperation with related organisations - Excellence of the project management plan to prevent delays in project implementation
Presentation (40%)	Expected effects and sustainability of the project	50	- Expected outcomes (solving local problems, spreading to other local governments, etc.) - Appropriateness and feasibility of quantitative KPIs - Excellence in post-project operation management plan - How to secure the sustainability of smart town creations

Third, Core Technology Discovery Projects are divided into ‘company-led’ types, targeting companies or universities with innovative technologies, and ‘citizen-led living lab’ types, led by local citizens, which solve urban problems. First, unlike previous local government support programs, ‘company-led’ projects aim to discover and demonstrate differentiated solutions among companies and universities that own innovative technologies. Administrative and financial support is provided to facilitate the implementation of innovative solutions, along with special procurement incentives such as the designation of innovative products with superior technology. The evaluation indicators for Core Technology Discovery Project comprised four written assessment items and three presentation evaluation items [27].

Table 4. 2022 Core Technology Discovery Project (company-led type) evaluation indicators.

Div.	Item		Score	Sub-items
Written assessment (60%)	Necessity and feasibility of the project		20	- Analysis of the current situation and problems at the target site and feasibility of solutions - Project goals and need of implementation
	Excellence and innovation of the technology		30	- Adequacy of solutions to achieve goals - Innovativeness of the technology/solution and its differentiation from existing solutions (economic and technological advantage, innovative utilisation, etc.) - Feasibility of the demonstration and adoption plan of the proposed solution - Specificity and achievability of quantitative KPIs
	Appropriateness of project direction		30	- Appropriateness of project governance system and collaboration arrangements - Specificity of citizen participation and stakeholder conflict control plans for each stage - Appropriateness of securing, linking, integrating, and utilising data collected by the solution - Specificity of sustainable promotion measures, such as reflecting in smart city planning
	Appropriateness of the budget use plan		20	- Specificity and appropriateness of project budget utilisation plans - Financing and investment plans of companies and local governments (labour cost accounting ratio, etc.)
Presentation (40%)	Commitment to the project implementation	Private actors’ willingness to commercialise	40	- Feasibility of the financing and investment plans of companies in implementing the project - Domestic and international expansion plans, including commercialisation of the solutions - Measures to promote sustainable smart cities, including the introduction of business models
		Local government commitment and feasibility	30	- Specificity of local government support and strategies to achieve goals - Feasibility of the local government’s budget and investment plan - Measures to ensure the sustainability of solution operations, and management and service provision
	Expected effects and sustainability		30	- Innovation, excellence, and sustainability of the proposed key solutions - Potential to comprehensively solve urban problems through the project - Economic and social effects of the proposed solution - Feasibility of project management plan for visible results - Measures to achieve socio-economic ripple effects such as enhancing regional competitiveness and job creation

‘Citizen-led living lab’ projects support companies and universities with innovative technologies or local governments that aim to demonstrate innovative technology by establishing and operating a sustainable smart city living lab tailored to each region. Companies and universities should clarify their target sites and collaborate with local governments and public institutions to provide administrative and financial support. Based on these characteristics, the evaluation indicators for citizen-led living lab type projects were identified as ‘Commitment to the project implementation’ and ‘Expected effects and sustainability of the project’, based on ‘Necessity and feasibility of the project’, ‘Excellence and innovativeness of the project’, ‘Living lab operation plan’, ‘Budget investment and utilisation plan’, and ‘sustainability’.

Table 5. 2022 Core Technology Discovery Project (citizen-led living lab type) evaluation indicators.

Div.	Item	Score	Sub-items
Written assessment (60%)	Necessity and feasibility of the project	20	- Necessity and goals of the project - Validity of target sites’ status and need analysis and the solutions - Local government cooperation (support) to advance and demonstrate innovative technologies
	Excellence and innovativeness of the project	20	- Excellence of innovative technology (solution) - Demonstration and advancement plan of innovative technology (solution) - Specificity and achievability of quantitative KPIs for advancement (problem solving) - Economic and social effects of the innovative technology (solution)
	Living lab operation plan	35	- Specificity and excellence of living lab organisational structure and collaborative governance - Living lab operational expertise and capabilities - Appropriateness of the living lab operation process - Excellence of business management plan to prevent delays in project implementation
	Budget investment and utilisation	15	- Company (university) investment plan - Appropriateness of budget utilisation (living lab operation, general expenses for innovation technology advancement, prototype production budget, demonstration budget, etc.)
	Sustainability	10	Specificity of post-project operation management plan (operation management plan for adopted solutions, cooperation plan for local governments, etc.) - Possibility of urban spread and expected effects
Presentation (40%)	Commitment to the project implementation	50	- Excellence and specificity of cooperation between companies/universities and local governments and project plan - Feasibility of financing and investment plans of companies/universities, etc. and overall business budget operation when promoting sustainable projects - Domestic and international spread plans such as commercialisation of innovative technologies (solutions) - Sustainable smart city promotion plan (e.g., introduction of business model) - Excellence of business management plan to prevent delays in project implementation
Presentation (40%)	Expected effects and sustainability of the project	50	- Expected outcomes (solving local problems, spreading to other local governments, etc.) - Appropriateness and feasibility of quantitative KPIs - Excellence in post-project operation management plan - How to secure sustainability

Finally, the Smart Campus Challenge Project was conducted to cultivate innovation, creativity, and problem-solving capabilities across the academy by providing participation opportunities for university students in smart cities and creating a boom. An additional function of the project was to discover and demonstrate innovative solutions through students’ creative ideas and link them to preliminary start-up commercialisation. Accordingly, the Smart Campus Challenge Project differs in many evaluation items from the Smart City-Smart Town Challenges, which aim to solve the current problems in cities and regions [28].

Table 6. 2021 Smart Campus Challenge evaluation indicators.

Div.	Item	Score	Sub-items
Prejudging (100)	Completeness of ideas	30	- Logics and validity of the conceptualisation of an idea and the development of a scientific theory, etc.
	Creativity and challenge	25	- Originality of a new method or design that is outside the existing method
	Differentiation	25	- Level of innovation or differentiation from existing or current practices
	Motive of suggestion	20	- Background and purpose of idea proposal - Relevance (fit) to smart cities and feasibility of solving urban problems
	Additional points	+4	- Non-metropolitan (up to 2 points) - Smart technology company (1 point) - Budding company (1 point)
Total		104

2.3. Smart City Policies by Country

Recently, the United Nations Framework Convention on Climate Change (UNFCCC) and the Paris Convention on Climate Change have been adopted to reduce greenhouse gas emissions, including carbon dioxide, and countries worldwide are striving to implement eco-friendly, high-efficiency energy systems. Smart-city-related policies are continuously being promoted to address this issue. Central governments actively promote technological development plans related to smart city construction [29].

The Smart City Challenge and Global City Team Challenge (GCTC) project is a representative smart city discovery demonstration project in the United States. The Smart City Challenge begins with discussions on smart solutions to solve urban problems with 15 government departments centred around the US Department of Transportation and promotes support projects by way of competition among the US cities. The program aims to build an innovation ecosystem network in the United States and worldwide. To this end, the US fosters start-ups that aim to expand to global companies and is expanding into a private-centred smart city industry that supports venture capital [30].

Singapore, an entire landmass occupied by urban spaces, has been promoting various policies for systematic urban management. In this context, a Smart Nation Plan has been established to respond to urban problems using digital technology. Singapore has promoted strong smart city policies for the systematic management of the city, including restricting vehicle ownership and securing green spaces in parks, to solve various city problems, such as low birth rates and aging populations, high population density, road congestion, and housing problems [31].

Japan’s population is expected to decrease by 20% from its current level. Despite the regional revitalisation that began in 2060 to maintain a population of 100 million, it is expected that it will be difficult to continue supplying essential city services in the future. Thus, a new smart city concept is needed to improve local quality of life using advanced technologies to realise a future society with resident consensus [32].

The Netherlands operates the Amsterdam Smart City (ASC), a public-private partnership (SPC) corporation, to solve urban problems by utilising the collective intelligence of various participants, including the government, businesses, academia, and citizens of Amsterdam. The ASC platform operates projects in six fields, with 2,000 employees, 90 partners, and over 100 projects in progress (as of 2020), including urban design, Artificial Intelligence (AI), and the Internet of things (IoT; establishing goals through smart convergence technologies such as the IoT), and is converted to a service model that provides Connectivity Digital Platform services using data.

China’s smart city policy began to be promoted in earnest in 2012, and the ‘New Urbanisation Plan’ was announced in 2014. A central government-centred policy is being promoted. Through the 13th Five-Year Plan of 2015, nationwide smart city construction was promoted until 2020, and as of 2020, the number of smart city pilot cities in China is estimated to be approximately 900. The direction of China’s smart city policy presented in the ‘New Urbanisation Plan’ is the six detailed directions of Δinformatisation of information network, Δinformatisation of planning management, Δintelligence of infrastructure, Δconvenience of public service, Δmodernisation of industrial development, and Δrefinement of social management. This plan is primarily related to information and communication technology [33].

Spain’s smart city national plan aims to support relevant industries, as well as local governments during the transition process, and promote reuse, standards, and interoperability. For the smart cityisation of local governments, support for ICT efficiency verification projects and the development of the ICT sector, standardisation, governance, and industry models are built, interoperability with the International Telecommunication Union (ITU), open data, tourist destination analysis, and rural environment citizen-centred services based on digital technologies commonly found in Spanish smart cities, such as developing and applying standardisation models with topics such as development and establishing a governance model involving all stakeholders, both public and private, through governance [34].

2.4. Research Methodology

2.4.1. Analytic Hierarchy Process

In this study, we used the AHP technique to determine the relative priority among elements constituting multiple hierarchies. AHP is a hierarchical decision-making technique that has been proven reliable [35]. The extent to which a measure fits its purpose is important because decision-making evaluates something intangible that needs to be traded off. An element for a given attribute which dominates another element is determined using a measure of absolute judgment. These judgments may be inconsistent, and AHP measures discrepancies to improve judgments where possible and obtain better consistency [36].

The basic problem of decision making involves choosing the best alternative with contradictory standards and incomplete information. The AHP provides a comprehensive framework to address these issues, allowing us to simultaneously address rational, intuitive, and irrational decisions when making multi-criteria and multi-agent decisions [37]. AHP can overcome the limitations of existing statistical analysis methods and simultaneously determine the relative importance of several criteria using a systematic and scientific method [38].

Vargas [39] proposed that the theoretical basis for the application of AHP is revealed by four axioms. First, reciprocal refers to reciprocity; the decision-maker must be able to pair and compare two factors within the same hierarchy. Furthermore, the strength of the preference must be expressible and satisfy reciprocal conditions. Second, the elements of one layer must depend on those of the upper layer. However, it is not necessary to ensure the independence between the elements in the lower hierarchy for all elements in the upper hierarchy. Third, expectations assume that the hierarchy includes matters related to decision-making. Fourth, through homogeneity, importance is expressed using a set scale within a limited range. These four axioms are necessary conditions for inducing validity in situations where the AHP is maintained or theoretically used in practice.

2.4.2. Efficiency and Productivity Measures

The Data Envelopment Analysis (DEA) technique is a nonprofit decision-making unit that performs the same or a similar function by utilising many inputs and outputs based on a system model. It is an efficiency measurement method that uses a non-parametric method to measure the relative managerial efficiency in the management operations of decision-making units (DMUs) [40].

This model has consistently been used as the most appropriate method for evaluating inefficiency in the public sector, where market prices are not formed [41]. The DEA model fixes one of the inputs and outputs and is divided into input-oriented and output-oriented models according to how inefficient parts are found for the remaining elements. Furthermore, depending on whether the effect of scale is considered when measuring efficiency, it can be divided into Charnes, Cooper and Rhodes (CCR), and Banker, Charnes and Cooper (BCC) models. The CCR model assumes constant returns to scale (CRS) for the production set, and the BCC model assumes variable returns to scale (VRS).

2.5. Literature Review

In this section, we examine the metrics related to smart cities used to derive important variables by considering the characteristics of this study. Regarding the AHP analysis, as well as efficiency and productivity analysis, a detailed review of previous studies is necessary because the selection of variables closely affects the reliability of the analysis results, including solidity and validity [42]. First, a study derived 54 indicators to constantly monitor the degree of smart city construction in global cities worldwide based on the EU’s six core factors: environment, mobility, economy, people, government, and living [43]. We examined this in detail to measure efficiency and productivity, because we recognised the need for companies to monitor and measure efficiency and productivity. Another analysis study based on 5 fields, 21 subfields, and 80 performance indicators to evaluate the governance, innovation, sustainability, connectivity, and cohesion of smart cities ensured diversity in the evaluation system [44]. Another study constructed non-technical categories of evaluation indicators, not only for construction purposes, but also to reduce operating costs and secure sustainability in the long run by building a model related to the maturity of smart cities. This implies that sustainability must be considered as well [45]. Some metrics are also organised according to the significance of performance outcomes. Since this is linked to performance indicators, this study identifies the key success factors that make projects work while adhering to the original plan [46]. Some studies derived indicators from urban perspectives. In one study, smart city indices were derived from environmental, economic, and social perspectives by various research institutes, which means that evaluation indicators can be used to select projects as well as to spread them across the city [47]. As smart cities are a convergence of digitalisation and technology, there is also an industry measurement scale, based on 10 areas and 40 indicators, to foster support for digital start-ups working on them [48]. Finally, another study ranked the top 20 global smart cities into four categories based on the performance of each city’s current operations and projects: mobility, health, safety, and productivity [49]. This indicates that smart cities are not only active in South Korea, but are also part of the global trend.

Furthermore, studies have been conducted on living labs and governance in which real residents participate to ensure sustainability. To solve urban problems using advanced technology, the problems experienced by residents in real life should be identified and solved. In this respect, to successfully promote smart city policies, it is necessary to establish a living lab and governance where citizen participation is essential [50]. Living labs support innovation by promoting existing policies, overcoming the limitations of government-led top-down promotion, and emphasising the reflection of local and citizen perspectives and public-private partnerships. As the application of living labs is expanding as a technology commercialisation and testbed for smart city experimentation and implementation, it is important to share the experience of living labs and build a network that can be applied as a new innovative model of the existing national space policy. A living lab is defined as a user-led innovation model that generally undergoes the stages of problem discovery, business planning, prototype design, product and service development, and demonstration. In this process, it develops into a win-win ecosystem in which various stakeholders exchange value at a reasonable level, premised on the continuity of service and securing a network of participants [51]. In particular, as citizens—the end users of the service—participate in the entire process of the project, continuous use and expansion of the service will be considered from the beginning of the project, which is consistent with the goal of creating a sustainable smart city. Overall, although the essential concepts of living labs and smart cities differ from the value chain viewpoint of means and ends, they complement each other in that they pursue the common goal of realising a sustainable city; these two complex application cases are expected to increase further in value. As discussed, studies on smart cities are extensive and diverse in scope and fields. However, most of these studies focused on the measurement of one or two metrics. Smart cities should be approached from the perspectives of convergence, service operations, software, IT evaluation, operational types, and sustainability. For these projects to be sustainable, it is necessary to develop more innovative and active projects, including private participation, rather than building infrastructure in the public sector, based on an understanding of the operation and management status of companies. This study is different from previous studies as it derives priorities by weighting convergence and convergence aspects to standardise the evaluation system, and it presents results to secure sustainability by analysing the efficiency and productivity of Korean smart city convergence alliance companies.

3. AHP analysis results

3.1. Organisation of Hierarchy Diagram

The hierarchy diagram was structured based on the evaluation indicators mentioned in the theoretical consideration section, while considering the sub-items of the metrics of previous studies. In particular, the metrics were reclassified to account for the scoring and items used to derive the tiers. Tier 1 is shown in Table 7.

Table 7. Evaluation system (Tier 1).

Smart Challenge Project				Evaluation items	Score	Reclassification(Tier 1)
City	Town	Company-led	Citizen-led living lab	Evaluation items	Score	Reclassification(Tier 1)
●	●	●	●	- Necessity and feasibility of the project	20	Technical expertise
		●		- Excellence and innovativeness of the innovative technology	30
			●	- Excellence and innovativeness of the project	20
			●	- Living lab operation plan	35
	●			- Excellence of the project plan	30	Specificity of planning
●				-Adequacy of the project implementation plan	30
		●		-Adequacy of the project promotion plan	30
		●		-Adequacy of budget utilisation plan	20
			●	-Investment and utilisation of budget	15
	●			- Sustainability of the project	20	Sustainability
●				-Cooperation with citizens and stakeholders	10
●				-Cooperation among participating organisations	20
●				-Specificity and excellence of the main project implementation plan	20
●	●	●	●	-Expected effects and sustainability of the project	30–50
●	●	●	●	- Willingness and feasibility of local governments	35–50	Scalability
●	●	●	●	- Willingness of private participants to commercialize	35–50	Scalability

Based on the Tier 1 results, we conducted five rounds of FGIs with a group of experts to reflect these results and assigned standardised weights to reflect the situation in South Korea, thus finalising the hierarchy diagram including Tiers 2 and 3 (Table 8).

Table 8. Final hierarchy diagram.

Goal	Comparison of importance (Tier 1)	Comparison of importance (Tier 2)	Comparison of importance (Tier 3)
Smart city evaluation indicators	Technical expertise	Technology acceptability	Possession of specialized ICT technicians, establishment of a dedicated smart city department, existence of interconnected services between departments, and use of big data in policy proposals by field.
	Technical expertise	Administrative accessibility	Organize smart city living labs and governance, facilitate citizen policy input, and computerize administrative services
	Plan specificity	Economic infrastructure	Build a virtuous employment ecosystem, open consumption behaviour, local productivity, and industry spillovers
		Urban infrastructure	Validity of building intelligent facilities, enterprise management system (EMS), sustainability of transportation facilities
		Educational infrastructure	Introduce specialised ICT training and E-learning, provide intelligent educational facilities
	Sustainability	Social infrastructure	Operate integrated operational centre and sustainable social council
		Living infrastructure	Smart city planning, smart healthcare and safety management
		Environmental infrastructure	Sustainability of environmental facilities, smart environmental management, energy management
	Scalability	Fostering specialised human resources	Technology convergence workforce education, regional networked innovative workforce
	Scalability	Creation of industrial ecosystem	Strengthen public-private/private-private partnerships capabilities, support for overseas export of innovative products

3.2. Analysis Result

The questionnaire was distributed to 100 experts working in smart-city-related fields. Of the 100 returned responses, 83 valid samples were used, after excluding inconsistent and incomplete responses (17). Respondents included 23 university professors, 20 civil servants, 17 general employees, 12 researchers, and 11 public companies. The AHP results, which maintained the independence of each tier are shown in Table 9.

Table 9. Final prioritisation results.

Evaluation Goal (Tier 1)	Evaluation Criteria [Main] (Tier 2)	Evaluation Criteria [Sub1] (Tier 3)	Evaluation Criteria [Sub2] (Tier 4)		Final Weight	Tier Ranking
Improvement of smart city project planning evaluation system	Technical expertise (L: 0.16)	Technical acceptability (L: 0.66)	Possession of ICT specialized technical personnel	(L: 0.24)	0.0253	15
			Establishment of a dedicated smart city department	(L: 0.38)	0.0401	8
			Existence of interconnected services between departments	(L: 0.22)	0.0232	18
			Utilisation of big data for policy proposals by field	(L: 0.16)	0.0169	25
		Administrative accessibility (L: 0.34)	Smart city living lab and governance organization	(L: 0.48)	0.0261	12
			Ease of reflecting citizen policy opinions	(L: 0.34)	0.0185	24
			Computerisation of administrative services	(L: 0.18)	0.0098	27
	Specificity of plan (L: 0.30)	Economic infrastructures (L: 0.29)	Building a virtuous employment ecosystem	(L: 0.30)	0.0261	12
			Open consumption behaviour	(L: 0.18)	0.0157	26
			Local productivity	(L: 0.23)	0.0200	23
			Industrial spillovers	(L: 0.29)	0.0252	16
		Urban infrastructures (L: 0.57)	Feasibility of building intelligent facilities	(L: 0.40)	0.0684	4
			Enterprise management system (EMS)	(L: 0.40)	0.0684	4
			Sustainability of transportation facilities	(L: 0.20)	0.0342	10
		Educational infrastructures (L: 0.14)	Introduce ICT professional training and E-learning	(L: 0.51)	0.0214	21
		Educational infrastructures (L: 0.14)	Provide intelligent educational facilities	(L: 0.49)	0.0206	22
	Sustainability (L: 0.33)	Social infrastructures (L: 0.41)	Operate an integrated operations centre	(L: 0.59)	0.0798	3
		Social infrastructures (L: 0.41)	Operate sustainable social councils	(L: 0.41)	0.0555	6
		Living infrastructures (L: 0.38)	Establish smart city plans	(L: 0.79)	0.0991	1
		Living infrastructures (L: 0.38)	Smart medical and safety management	(L: 0.21)	0.0263	11
		Environmental infrastructures (L: 0.21)	Sustainability of environmental facilities	(L: 0.36)	0.0249	17
			Smart environmental management	(L: 0.33)	0.0229	19
			Energy management	(L: 0.31)	0.0215	20
	Scalability (L: 0.21)	Nurture professional workforce (L: 0.31)	Technology convergence workforce education	(L: 0.61)	0.0397	9
		Nurture professional workforce (L: 0.31)	Regional networked innovative workforce	(L: 0.39)	0.0254	14
		Creation of industrial ecosystems (L: 0.69)	Strengthening public-private/private-private partnerships	(L: 0.68)	0.0985	2
		Creation of industrial ecosystems (L: 0.69)	Support for overseas export of innovative products	(L: 0.32)	0.0464	7

From the AHP, smart city planning (0.9991) ranked first, followed by public-private/private-private partnership capacity building (0.0985), integrated operation centres (0.0798), feasibility of building intelligent facilities (0.0684), and EMS (0.0684). This analysis reflects the recent trends of all local governments in Korea regarding the establishment of smart city plans. Therefore, it is desirable to promote smart city projects with a long-term sustainability plan, rather than as a one-off event. Public-private/private-private partnership capabilities should be strengthened to ensure project scalability. Ultimately, securing profitability and stability seems impossible when the private sector participates in and leads projects. The feasibility of building intelligent facilities also seems to reflect the expert opinion that these facilities are required to address sustainability issues. Ultimately, a smart city project requires a substantial investment, and it is important to ensure cost-effectiveness. It appears that feasibility should be secured in advance, and there may be frequent failures and significant maintenance costs owing to the nature of electronics and devices. Finally, EMS is considered highly important because of the characteristics of a smart city project, which requires integration and effective control of various subjects and departments, and smooth analysis and utilisation of the database.

4. Efficiency and Productivity Analysis Results

In addition to the results of the previous analysis, we conducted an efficiency and productivity analysis on 38 SMEs that are the most sensitive to business performance among the currently operating smart city convergence alliance participants, to use as the basis for an evaluation system. The analysis period was limited to three years, from 2019 to 2021, and an analysis model was established to identify trend changes over the three years. The variables used in the analysis were capital (KRW million), number of employees (people), assets (KRW million), sales (KRW million), operating income (KRW million), and net income (KRW million). Regarding the variables, the items generally used to evaluate the management of companies were prioritised as the scope that can measure the actual management status. Although additional variables can be reflected, the analysis of efficiency and productivity can be unreliable when the number of variables is more than twice that of the DMUs. Therefore, they were not included to ensure the validity and reliability of the analytical model. The basic statistics are shown in Table 10.

Table 10. Basic statistics.

Year	Category	Minimum	Maximum	Mean	SD
2019	Capital (KRW million)	567	148,768	11,309.50	24,912.63
	Number of employees (people)	1	268	54.95	71.50
	Assets (KRW million)	832	235,774	18,564.68	40,117.95
	Sales (KRW million)	122	94,172	12,770.98	18,872.19
	Operating income (KRW million)	-6,521	5,985	78.50	1,759.19
	Net income (KRW million)	-29,524	5,758	-1,000.20	5,231.20
2020	Capital (KRW million)	637	173,417	11,933.95	28,302.18
	Number of employees (people)	1	285	62.60	75.56
	Assets (KRW million)	951	251,398	20,486.45	41,861.79
	Sales (KRW million)	77	91,237	15,187.32	21,907.16
	Operating income (KRW million)	-6499	6,576	446.50	2,150.39
	Net income (KRW million)	-17139	7,256	55.90	3,405.35
2021	Capital (KRW million)	699	200,921	15,099.75	33,297.94
	Number of employees (people)	6	375	76.50	86.58
	Assets (KRW million)	1,054	260,651	23,762.50	45,064.57
	Sales (KRW million)	63	82,219	14,994.60	20,483.75
	Operating income (KRW million)	-7,488	25,486	968.87	4,582.89
	Net income (KRW million)	-9,418	19,066	810.40	4,029.82

Particular attention should be paid to operating costs and net income. The maximum annual value follows an upward trend. In particular, the figure will be approximately three times higher in 2021 than in the previous year. This implies that smart city projects will enter a steady state with increasing income from 2021 onwards. However, they are still in the red despite the decrease in the minimum value; therefore, it is urgent to address this problem through technological innovation and policies that temporarily relieve regulations to demonstrate smart city projects, among others.

Table 11. CCR analysis results.

DMU	2019	2020	2021
1	0.4782	0.6321	0.6951
2	1	1	1
3	1	1	1
4	1	1	1
5	0.6852	1	0.9785
6	1	1	0.9213
7	1	1	0.9921
8	1	1	1
9	0.8212	0.8631	0.8323
10	0.5733	0.5752	0.6845
11	0.632	0.4302	1
12	0.7732	0.9561	0.9429
13	1	0.9212	0.9124
14	0.8633	0.6752	0.3829
15	0.4469	0.4932	0.2764
16	0.8424	1	1
17	0.8358	0.6621	0.6348
18	0.8902	0.9631	1
19	0.6128	1	1
20	0.6332	0.4921	0.9212
21	0.9562	0.9102	0.9212
22	0.7632	0.7921	1
23	0.8236	0.6982	0.7823
24	0.9563	0.6933	0.6952
25	0.921	0.9215	1
26	0.3452	0.5231	0.6218
27	0.6213	0.5212	0.1625
28	0.5852	0.9612	0.5289
29	0.5775	0.6907	0.6619
30	1	0.584	0.7113
31	1	1	0.9932
32	0.5992	0.5551	0.6212
33	0.6132	0.4822	0.4232
34	0.6333	0.5212	0.3921
35	1	0.9212	0.8218
36	0.5236	0.4921	0.4922
37	0.792	1	1
38	0.5632	0.6341	0.7521
Average	0.7727	0.7780	0.7830

The CCR analysis results show a slight increase in efficiency from an average of 0.7727 in 2019 to 0.7830 in 2021. For each unit, DMUs 2, 3, 4, 6, 7, 8, 16, 19, 31, and 37 had a value of 1 or close to 1, indicating that they were operationally efficient. In the future, it will be necessary to select and share best practices by monitoring and consulting with these companies to identify policy factors that can help them improve their operational efficiency.

Table 12 shows the results of the BCC analysis, categorised into technical efficiency (TE), pure technical efficiency (PTE), and scale efficiency (SE). TE was 0.8102, PTE and SE were 0.0973 and 0.8642, respectively; when the value is close to 1, it is considered efficient. Under the CRS and VRS assumptions, smart city convergence alliance SMEs were found to maximise their efficiency by reducing their inputs by 81% and 86%, respectively. As all values are on the upward curve from year to year, operational efficiency is expected to improve gradually.

Table 12. BCC analysis results.

Category	2019	2020	2021	Average
TE	0.7843	0.7932	0.853	0.8102
PTE	0.8671	0.8522	0.8732	0.8642
SE	0.8782	0.9135	0.9301	0.9073
N	38	38	38	38

TE: technical efficiency; PTE: pure technical efficiency; SE: scale efficiency; N:.

Table 13. Analysis of return to scale.

Category	2019	2020	2021	Sum
CRS	9	11	11	31(27.19%)
DRS	11	10	12	33(28.94%)
IRS	18	17	15	50(43.87%)
N	38	38	38	114(100%)

CRS: constant returns to scale; DRS: ;IRS: ; N:.

Finally, the three-year time-series average productivity index are shown in Table 14.

Table 14. Malmquist productivity index change rate.

Time-series	TECI	TCI	PECI	SECI	MPI
T2	0.9893	1.0555	0.9526	1.0386	1.0442
T3	0.9538	0.9482	0.9339	1.0213	0.9044
Geometric mean	0.9714	1.0004	0.9432	1.0299	0.9718

TECI: ;TCI: ; PECI: ; SECI: ; MPI: municipal performative index; T2: ; T3:

Finally, for productivity analysis, the degree of change between the two time points was measured. If the municipal performative index (MPI) value of the time series at T2 and T3 is greater than 1, this indicates a mutual increase in productivity at two separate points in time, t and t+1, from 2019 to 2020 and 2020 to 2021, respectively. In other words, when the value is greater than 1, productivity increases; when it is 1, there is no change between the time points; and when it is less than 1, productivity decreases. After analysis, T2 showed an increase in productivity, whereas T3 showed a decrease. Taken together, these efficiency results suggest that the problem is more operational than technical. Some companies have technology that cannot be demonstrated because of legal and institutional barriers, and recently, external factors such as inflation, high interest rates, and labour shortages also seem to be involved.

5. Discussion

5.1. Summary and Significance of the Study

This study identifies, analyses, and prioritises the significant factors for standardising the Korean smart city project evaluation system. In addition, we analyse the operational performance of companies currently providing smart city services in terms of efficiency and productivity to consider both policy and practical aspects. This is a timely study, as a smart city project is not a one-time project but directly affects the quality of life of local residents. Furthermore, given the characteristics of smart city projects that require substantial resources and that are implemented by several departments simultaneously, studying the standardisation of the evaluation system is necessary. Most countries, including developing countries, are carrying out smart city initiatives. The development of global standards should also be discussed. In this context, we examined the evaluation factors and elements derived from previous studies and smart city evaluations conducted in South Korea, based on which a hierarchical diagram was constructed.

Twenty-seven major factors were derived and confirmed through five rounds of FGIs, and with experts identifying the priorities. In addition, we conducted an efficiency and productivity analysis of 38 SMEs involved in smart city convergence alliances and used them as the basis for an evaluation system. The analysis is not limited to a short-term perspective but presents both individual efficiency and overall productivity analysis results over time. This study is expected to serve as the basis for building a standardised evaluation system that includes both policy and practical implications for companies. The AHP showed that smart city planning (0.9991) ranked first, followed by public-private/private-private partnerships capacity building (0.0985), integrated operation centres (0.0798), feasibility of building intelligent facilities (0.0684), and EMS (0.0684). This result reflects that the eligibility requirement for local governments to apply for the Korean Smart City Project contests was specified as the establishment of a smart city plan rather than a general city plan. In some cases, only additional points were given; however, recently, all local governments have been establishing smart city plans, and this analysis seems to reflect this trend. Therefore, it is desirable to promote smart city projects with a long-term sustainable plan rather than as a one-off event, as indicated by the analysis results. Public-private/private-private partnership capabilities should be strengthened to ensure project scalability. Until now, the public sector has led smart city projects to support the basic infrastructure. However, this has had its side effects, such as inconsistencies in projects, difficulty with the introduction of innovative technologies, and one-off election-promoting projects. Ultimately, projects led by the private sector seem more stable. The feasibility of building intelligent facilities also reflects the expert opinion that it is necessary to address the issue of sustainability. Ultimately, a smart city project requires huge investments, and it is important to ensure cost-effectiveness. Feasibility should be secured in advance, as there may be frequent failures and significant maintenance costs owing to the nature of electronics and devices. Finally, an EMS is deemed highly important because of the characteristics of a smart city project, which requires integration and effective control of various subjects and departments, and smooth analysis and utilisation of the database. In addition to the previous analysis results, we conducted an efficiency and productivity analysis on 38 SMEs that are the most sensitive to business performance among the currently operating smart city convergence alliance participants and used them as the basis for an evaluation system. The analysis period was limited to three years from 2019 to 2021, and an analysis model was established to identify trend changes over the three years. According to the analysis results, under the CRS and VRS assumptions, smart city convergence alliance SMEs can maximise their efficiency by reducing their input by 81% and 86%, respectively. However, as all values are on the upward curve from year to year; the operational efficiency is expected to gradually improve.

The implications and policy recommendations derived from these results are as follows. First, to improve the smart city project plan evaluation system, project evaluation, and to revitalise the smart city industry, it is necessary to supplement the evaluation index by considering the relative importance of the indices to ensure that the sustainability of projects can be evaluated. Furthermore, because the purpose of the Smart Challenge project is to utilise corporate solutions based on the ideas of universities and citizens regarding urban problems, evaluation indicators for companies must be considered for the smooth promotion of the project.

Second, based on the smart city planning of local governments, standard guidelines were prepared to promote business expertise, business plan specificity, and business scalability, and among companies with smart city technologies, to expand specialised technologies for SMEs. To secure a new market, institutional support, such as financial support for commercialisation from central and local governments, tax benefits, incentives, and living labs involving residents, should be strengthened along with expert training.

Third, the results indicate that smart city-related companies in Korea improve productivity by increasing efficiency through operational rather than technological improvements. Therefore, innovative technologies such as digital twins, platforms, IoT, AI, big data, and Geographic Information Systems/ Global Positioning Systems are advanced, and joint businesses are strengthened, while the legal and institutional aspects of actual operations are flexible as the basis for securing profitability. Policy alternatives are required, such as the preparation and activation of policies that temporarily relieve regulations to demonstrate smart city projects.

5.2. Research Limitations and Future Tasks

This study had some limitations. Numerous smart city projects should not be viewed simply, but from multiple perspectives; therefore, their purpose and direction are not always clearly aligned. In addition, although we set up a research model in several ways and presented each analysis result, we could not score or standardise them. Furthermore, AHP, efficiency, and productivity analyses are based on qualitative and quantitative data, respectively; therefore, applying the same point in time to the same variable is problematic. In particular, in efficiency and productivity analyses, the failure to classify the types of smart city convergence alliance companies should be addressed. Future researchers could benefit from providing a three-dimensional perspective by addressing the limitations of this study.

References

Myeong, S.; Jung, Y.; Lee, E. A study on determinant factors in smart city development: An analytic hierarchy process analysis. Sustainability 2018, 10, 2606. [Google Scholar] [CrossRef]
Telecommunications Technology Association. Key convergence cases of the 4th industrial revolution smart city concept and standardization status, standardization issue 2018-1, 2018. Available online: URL (accessed on Day Month Year).
Seoul City Council. A study on analyzing and solving social problems in the digital city of Seoul, 2018. Available online: URL (Accessed on Day Month Year).
Peter Hansen, L. Repercussions of pandemics on markets and policy. Rev. Asset Pricing Stud. 2020, 10, 569–573. [Google Scholar] [CrossRef]
Belanche, D.; Casaló, L.V.; Orús, C. City attachment and use of urban services: Benefits for smart cities. Cities 2016, 50, 75–81. [Google Scholar] [CrossRef]
Hwang, J.; Jang, J. Prospects for smart city development and Korea’s competitiveness. IT & Future Strategy. 2016 No. 6; National Information Society: Daegu Agency.
Kim, Y.; Koo, J. Study on the developing of evaluation indicators for smart city from the perspective of digital social innovation. J. Korea Contents Assoc. 2019, 19, 512–521. [Google Scholar]
Han, S.; Shin, Y.; Yu, I.; Lee, J. A study on the Korea smart city certification index and demonstration authentication. J. Korea Acad.-Ind. Coop. Soc. 2018, 19, 688–698. [Google Scholar]
Ahn, Y.; Lee, S.; Yu, M.; Jeong, G.; Yeom, I.; Ji, N.; Kim, A. Establishment of a smart city model based on citizen participation; Daejeon Sejong Institute: Daejeon, 2018. [Google Scholar]
Jin, S. Life satisfaction depending on digital utilization divide within people with disabilities. Informatization Policy 2019, 26, 69–89. [Google Scholar]
Ministry of Land, Infrastructure and Transport, 2022. Smart city industry promotion strategy research final report. Available online: URL (accessed on Day Month Year).
Dutton, W.; Blumler, J.; Kraemer, K. Wired cities: Shaping the future of communications; G.K. Hall: New York, USA, 1987. [Google Scholar]
Ishida, T. Understanding digital cities. In Digital cities: Experiences, technologies and future perspectives lecture notes in computer science; Ishida, T., Isbister, K., Eds.; 1765. Springer-Verlag: New York, USA, 2000. [Google Scholar]
Aurigi, A. Competing urban visions and the shaping of the digital city. Know. Techn. Pol. 2005, 18, 12–26. [Google Scholar] [CrossRef]
Mora, L.; Bolici, R.; Deakin, M. The first two decades of Smart-city research: A bibliometric analysis. J. Urban Technol. 2017, 24, 3–27. [Google Scholar] [CrossRef]
Yigitcanlar, T.; Kamruzzaman, M.; Foth, M.; Sabatini-Marques, J.; da Costa, E.; Ioppolo, G. Can cities become smart without being sustainable? A systematic review of the literature. Sustain. Cities Soc. 2019, 45, 348–365. [Google Scholar] [CrossRef]
OECD. Smart cities and inclusive growth; OECD: Paris, 2020; Available online: URL (Accessed on Day Month Year). [Google Scholar]
ITU-T, 2014. Smart sustainable cities: An analysis of definitions [ITU-T Focus Group on Smart Sustainable Cities Technical Report]. Available online: URL (Accessed on Day Month Year).
Cohen, B. The 3 generations of smart cities: Inside the development of the technology driven city, 2015. (accessed on Day Month Year). Available online: https://www.fastcompany.com/3047795/the-3-generations –of-smart-cities.
Nam, T.; Pardo, T.A. Conceptualizing smart city with dimensions of technology, people, and institutions, The 12th Annual International Digital Government Research Conference: Digital Government Innovation in Challenging Times, College Park Maryland, USA, 2011.
Kang, D.; Choi, C.; Lee, J.; Lee, J.S.; Park, S. Employment impact assessment study for the global smart city demonstration complex project; Korea Labor Institute: Sejong, 2016. [Google Scholar]
Karvonen, A.; Cugurullo, F.; Caprotti, F. Inside smart cities: Place, politics and urban innovation; Routledge: London, 2019. [Google Scholar]
Eip, S.C.C., 2018. European innovation partnership on smart cities and communities, General Assembly 2018 Summary Report, Sofia, Bulgaria. Available online: URL (accessed on Day Month Year).
Jung, S. Policy trends for smart city promotion in Europe. Technol. Trends Wkly 2019, 1921, 2–13. [Google Scholar]
Ministry of Land, Infrastructure and Transport 2021. Announcement of the 2021 smart city challenge project competition. Available online: URL (accessed on Day Month Year).
Ministry of Land, Infrastructure and Transport 2021. Announcement of the 2021 smart town challenge project contest. Available online: URL (accessed on Day Month Year).
Ministry of Land, Infrastructure and Transport 2022. Announcement of competition for smart city innovative technology discovery project in 2022. Available online: URL (accessed on Day Month Year).
Ministry of Land, Infrastructure and Transport 2022. Announcement of competition for regional base and small city smart city creation project in 2022. Available online: URL (accessed on Day Month Year).
Kim, K.; Kim, G. Status and prospect of smart city in the fourth Industrial Revolution. J. Korean Converg. Soc. 2018, 9, 191–197. [Google Scholar]
Lee, J.; Lee, M.; Lee, J.; Kim, I. Strategic response measures according to smart city types, Korea Research Institute for Human Settlement, 2018.
Cho, J.; Lee, N.; S.; W. Smart city overseas case study in Singapore, smart city policy and governance research, Seoul Digital Foundation, 2021.
Cho, M.; Lee, J., 2021. Contents and implications of the Japanese super city initiative to resolve resident-oriented local challenges, National land issue [Report]; Volume 42.
Lee, H.; Bang, S.; Kim; Oh, H.; Wang, F. China smart city promotion status and entry strategies: Focusing on the cases of Xiong’anjin-gu and Tianjin Eco-city; Korea Institute for International Economic Policy and Korea Research Institute for Human Settlements, 2021.
Korea Research Institute for Human Settlements; Smart City Association, 2022. Smart City Policy Forum.
Saaty, T.L. Rank generation, preservation, and reversal in the analytic hierarchy process. Decis. Sci. 1987, 18, 157–177. [Google Scholar] [CrossRef]
Saaty, T.L. Decision making with the analytic hierarchy process. Int. J. Serv. Sci. 2008, 1, 83–98. [Google Scholar] [CrossRef]
Saaty, T.L. Axiomatic foundation of the analytic hierarchy process. Manage Sci 1986, 32, 841–855. [Google Scholar] [CrossRef]
Woo, C.; Kim, G.; Kang, S. Comparative study of bankruptcy prediction models using LOGIT analysis and AHP analysis. Financ. Manag. Res. 1997, 14, 229–252. [Google Scholar]
Vargas, L.G. An overview of the analytic hierarchy process and its applications. Eur. J. Oper. Res. 1990, 48, 2–8. [Google Scholar] [CrossRef]
Charnes, A.; Cooper, W.W.; Rhodes, E. Measuring the efficiency of decision making units. Eur. J. Oper. Res. 1978, 2, 429–444. [Google Scholar] [CrossRef]
Kim, S. Structure analysis of production efficiency of local public service supply, Korean. J. Local Auton. 2000, 12, 47–65. [Google Scholar]
Shim, H.; Kim, J. Measures for the improvement of feasibility studies and investment review: Identification and verification of major project sectors considering balanced regional development. Sustainability 2020, 12, 9531. [Google Scholar] [CrossRef]
Smart Cities Council. Smart cities index. Available online: URL (accessed on Day Month Year), 2011.
INTELI. Towards a smart cities index: The case of Portugal. Available online: URL (accessed on Day Month Year), 2012.
IDC. Smart cities and the Internet of everything: The foundation for delivering next-generation citizen services. Available online: URL (accessed on Day Month Year), 2013.
ITU-T. Global smart sustainable city. Available online: URL (accessed on Day Month Year), 2016.
EU city keys, 2017. Co-funded by the European Commission within the H2020 Programme. Available online: URL (accessed on Day Month Year).
Nesta. What next for digital social innovation. Available online: URL (accessed on Day Month Year), 2017.
Juniper research, 2017. Smart cities – What’s in it for citizens? Available online: URL (accessed on Day Month Year).
Chang, H. Study on issues and perception changes in smart cities: Focusing on news, blogs, and twitter data, cadastral and territorial. Information 2019, 49, 67–82. [Google Scholar]
Seong, J.; Park, I. ICT Living Lab case analysis and implications as a user-led innovation model. Sci. Technol. Research 2015, 15, 245–278. [Google Scholar]

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

© 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

Copyright: This open access article is published under a Creative Commons CC BY 4.0 license, which permit the free download, distribution, and reuse, provided that the author and preprint are cited in any reuse.