Sustainable Healthcare Systems: How Close Are We? A Multi-Phase Mixed-Methods Evidence Synthesis

1. Introduction

Healthcare is a sector that experiences surging progress the last decade, especially with the wider adoption of technology (it is enough to look at the annual numbers of new interventions on clinical trial in the USA [1] - 42966 in 2025 - to realize the pace of productivity).

However, there is significant fragmentation, in multiple components of the HCS, which became particularly evident, along with its impact, during the covid-19 pandemic, where the capacity of most healthcare centres was significantly tested.

Nevertheless, not all healthcare organizations “suffer” the same, during crisis, due to their different performance level. Hence, knowing their performance, against their short-to-mid-term and long-term objectives, is not only essential for the continuation of their operation, but also an advantage that they can turn into an opportunity.

Despite multiple efforts, continuity in healthcare is significantly hindered by systemic issues, associated with the governance of healthcare centres, their administration and their collaboration with other sectors (3 major components in WHO frameworks [2]), that are hard to monitor, they are rarely monitored and captured, and hence they remain unsolved.

Some examples of such systemic issues are: misaligned HCS components (e.g., ministries, insurers etc.) [3,4,5], the lack of accountability over the whole patient journey [5,6,7,8], misaligned incentives [8,9,10,11] (providers paid for volume, not for service continuity) and supply chain disturbances [12,13,14,15,16]. Also, diseases with comorbidities often cause significant disruptions in the HCS [7,17,18,19,20].

Given the importance of operational continuity in healthcare, this study investigates the spectrum of monitoring mechanisms for sustainable performance, applied today.

2. Materials and Methods

Study Design 

For addressing the questions of the study, an Integrated Multi-Phase Evidence Synthesis was conducted, with 3 phases, followed iteratively, including:

- A Systematic Review, following relevant parts of the PRISMA reporting guideline [21]

- A Scoping Review, adhering to the PRISMA-ScR reporting protocol [22]

- And an Evidence Synthesis, synthesizing the outcomes of the previous phases, by following the ENTREQ Equator-Network guideline [23].

Research Methodology 

The Systematic Review [Suppl1] aimed at building an understanding of the MFWs for healthcare centres, suggested by research recently, by reviewing peer-reviewed publications and further characterizing the MFWs they described, using a categorization framework – developed for this study-, against their capacity to help organizations operate sustainably.

As most reports’ findings were narrow and didn’t expand beyond the experimentation phases, for capturing a wider spectrum of monitoring approaches and for acquiring feedback of their true effectiveness in practice, a Scoping Review [Suppl2] was conducted.

In the Scoping Review, the monitoring approaches of 7 high-performing and 7 low-performing healthcare centres, according to CMS-ratings [24,25,26], were analysed, using grey literature and further characterized, under the same categorization framework, against their properties.

The use of the same categorization framework in both studies allowed, during the evidence synthesis phase, for the meaningful summarization of their findings, the discovery of common patterns, between the research methodologies and real monitoring practice and the identification of properties of the categorization framework that correlate with the differentiation in the organizations’ performance.

All phases of the study were visited iteratively, for reducing human error and until a robust conceptual framework for the evaluation and development of monitoring systems was shaped.

The Categorization Framework 

A categorization framework was built, tailored to evaluate each valid MFW’s capacity to enable healthcare delivery organizations, become more sustainable.

It is the product of the author’s prior research work and its validation against the 4Cs framework, presented by Serafeim G. [27,28,29,30].

The 4Cs framework highlights the major barriers, preventing organizations reach the performance frontier in their industry. The fundamental questions the 4Cs framework calls organizations to address, for improving their performances are: Clarity - do you see the performance gap from the best performance, and do you know your priorities? Craft - knowing the gap, do you know what to do and how? Consequences - knowing the gap and what to do next, are people

incentivized to do right thing? Collaboration - being anything in place, can units move together without friction [27,28,29,30]?

The categorization framework is composed of categories and subcategories, which are briefly described in Suppl3 and explained in detail in Table 1.

Information and Evidence Sources 

The Information Sources, Search Strategy, Screening and Study Selection, the Critical Appraisal, the Data Extraction and Categorization and the Descriptive Summary Statistics of the phase studies are summarized in Table 2. For more details, as well as, for the PRISMA and PRISMA-ScR diagrams, please visit Suppl1 and Suppl2.

Synthesis 

At this stage, a sufficiently broad view of the recent applicable monitoring approaches has been established, included of their effectiveness, in propelling the organizations sustainable performance, and of their shortcomings.

Furthermore, the qualitative difference, between the monitoring approaches of the two groups of organizations, provides real feedback on the monitoring dimensions that separate the two groups of performance.

The potential alignment of these properties, with the categorization framework, would further validate the framework, as an evaluation mechanism for MFWs.

Hence, the Synthesis phase provides a quite comprehensive response on how monitoring is performed and the mechanism to evaluate its effectiveness in healthcare organizations’ sustainable performance.

Meta-aggregation, critical-interpretative synthesis, deductive and inductive derivation of themes/categories and framework synthesis were applied in this phase.

3. Results

The main findings from the Systematic Review [Suppl1] are presented below:

3.1. Synthesis of the Systematic Review’s Findings

From 138 studies screened, only a 14% were describing a valid MFW, matching the study selection criteria (Table 2, Suppl1). This finding agrees with Zurynski et al.’s [31] and Khalil et al.’s [32] observation that healthcare systems’ performance conceptualization, definition and measurement are relatively unexplored, based on their study, prior to 2022.

Monitoring Approach 

39% of the frameworks proposed are extensions of long-established conceptual frameworks, such as the Donabedian [33], the TQM [34], Balanced Scorecards (BSCs) [35] and others, to the medical domain (Figure 1-Category). 33% used Conceptualization of Indicators - involving the statistical analysis of various forms of data, for the detection or direct selection of features, correlating with the constructs being monitored. 22% are technology-based, leveraging AI, Blockchain, Internet of Medical Things (IoMT) and Business Intelligence, over a reference architecture specific to the business domain of the environment being monitored.

Sustainability 

61% of the frameworks monitor either short-to-mid-term or long-term sustainability and 39% monitor both (Figure 1-Approach to Sustainability). 82% of the studies made no reference to any major sustainability governance framework, revealing that alignment with frameworks of international/national scale isn’t broadly considered a priority, during the development of MFWs.

Non-operational Function 

78% of the frameworks rely on iterative cycles with manual procedures, in the development and/or operation phase; thus, increasing the risk of unreliable and biased performance estimates.

Methodology 

50% of the studies apply scoring-based methodologies (Figure 1-Scoring vs Measuring), 17% measure-based, 17% are combining both methodologies and the rest are using other kinds of methods.

Scoring-based methods are cheaper to develop, require periodic human reviews and are considered less reliable than measure-based methods, which are most costly to implement, but generally cheaper to maintain and more interpretable [36,37].

Adoption of Technology 

44% operate on a digital platform, 33% can or already leverage AI and Data Analytics and 22% are fully automated.

Scalability 

83% of the frameworks can’t scale, decreasing the possibility of operating cross-border and cross-sector.

Risk 

28% of the performance evaluators, proposed in the studies, quantify the risk of missing targets. These frameworks use mostly reference-architecture monitoring approaches.

From the above findings, 2 broad categories of research MFWs emerge:

The first, where most of the frameworks belong, is characterized by the use of traditional techniques of monitoring (Extension-based, Conceptualization of Indicators), bearing legacy features, such as semi-automation, partial incorporation of sustainability, strong reliance on expert and stakeholder groups, limited adoption of AI and Data Analytics, limited scalability and so on. Though financially more attractive, they often lead organizations to a false view of where exactly they stand against their mission and to ratings non-functional, beyond the organization boundaries, preventing collaboration.

The second is characterized by technology-based frameworks, leveraging AI & other recent technologies, being often part of a broader telemonitoring hospital care model. Though the frameworks of this category exhibit optimal monitoring features, they are slowly or poorly adopted, due to their limited integration, with the healthcare environment [38,39,40,41].

3.2. Synthesis of the Scoping Review’s Findings

For a detailed view of the Scoping Review’s findings, please see Suppl2.

Insights from the high-performing healthcare centres 

The healthcare delivery organizations of this group, been rated in the top tier of quality by CMS [24,25], are multi-hospital, academic, medical centres, with millions of encounters annually and most cases reaching them being very complex [42,43]. Their revenue-base is diversified including patient services, research grants, philanthropic contributions, commercialization of technology and consultation to other organizations, amounting to several billion dollars, annually. Their operating margins are positive, but thin, and net asset change, although fluctuating, it is sustained across time, driven by investment return, philanthropy and retained earnings [Suppl4, 44]. They have good and sustained performance, however they are challenged in fulfilling their goals and vision across time.

Monitoring Approach 

Their approach is a mixture of monitoring systems, including BSCs, adapted to the medical domain, for internal governance reporting [35,45], CMS Conditions of Participation (CoP) [46] reporting, state authorities reporting, adhering to national and international accreditation organizations’ rules [47,48], tools for conforming with national and

academic benchmarks [49,50,51,52,53,54,55], with NSQA/HEDIS requirements for qualitative reporting, and last but not least, Internal Quality Systems (IQS), sustained significantly by digital platforms [56,57,58].

Their approach is based on all three types identified in the Systematic Review [Suppl1] and is largely driven by regulation. Key differences between organizations in this group come from the IQSs they implement (Table 3).

Sustainability 

They monitor both short-to-mid-term and long-term sustainability. Complying with CMS and local policy builds baseline capacity in these organizations and determines their eligibility to federal billing programs, thus impacting their long-term sustainability. Complying to accreditation organizations’ standards and operating IQSs, help them operate both with immediate resiliency and sustainably [47,48,56,57,58].

Non-operational function 

Although their performance is substantially monitored by their IQSs and their digital platforms [56,57,58], they are subject to audits and inspections by CMS, state and accreditation organization officials, which introduce manual procedures and stakeholders’ involvement, in the monitoring process [Suppl2].

Methodology 

As we don’t have access to the built-in methodologies of their IQSs, though these are mostly drawn from research [56,57,58], we focus on CMS’s methodologies, which are like those applied by other governance bodies, in their principles.

Most CMS Scoring methodologies [Suppl2,49–54], are influenced by research and they rely, on a combination of measure-based and index-based techniques [Suppl2,36,37].

Adoption of Technology 

They rely substantially on technology and on digital IQSs, for staying on the performance frontier, in medical services and innovation [68]. They spend at least 5% of their revenue annually on IT [Suppl4,59] and the amount gets higher, if including the investments on internal IT projects [Suppl4].

Scalability 

Performance evaluations are meaningful across country borders, mainly within their ecosystem, through their IQSs and their uniform governance system across centres, in multiple geographical sites [Suppl2].

CMS only partially integrates with international standards.

Risk 

Most IQSs [56,57,58] are equipped with performance optimization components that quantify and minimize the risk of an organization missing its internal and regulatory targets.

Insights from the low-performing healthcare centres 

There is an estimated 31% [60], according to CMS Overall Hospital Star Ratings [54], that are performing low, represented mostly by rural and/or safety-net hospitals and which are struggling to sustain their operations.

Their services are mainly of low-complexity, essential services, serving mostly uninsured and rural population. Many have been forced to close departments, due to lack of demand in commercial care and operational stability. Their revenue-base is constrained to patient services, and it amounts to tens of million dollars or less [Suppl4].

Monitoring Approach 

They monitor performance, using a composite model analogously with the high-performing healthcare centres, but with the difference of adhering also to local Certificate of Need [61,67] rules - in certain cases and only in some states - to accreditation bodies’ requirements [Suppl2], where applicable, and that their annual budget doesn’t allow for investing in IQSs.

Sustainability 

As they don’t invest in IQSs, they mainly monitor long-term sustainability, essentially trying to sustain their operations under razor thin margins, while staying compliant to CMS.

Non-operational Function 

As their monitoring relies mainly on CMS and local-state reporting, much of their performance monitoring is mediated by audits and inspections that CMS and local-states conduct quarterly or annually.

Methodology 

As complying to CMS and state regulation is their primary performance monitoring mechanism, their methodology follows a combination of measures’ monitoring and aggregation [Suppl2].

Adoption of Technology 

Due to their limited annual budget, their investment in technology is insignificant, hence their monitoring relies on CMS’s and states’ online platforms [62,63].

Scalability 

Any performance rating they preserve, only partially or not at all, integrate with ratings of other, cross-border or national, units.

Risk 

Although they usually don’t maintain IQSs, some medical centres are enforced by state regulation [64], to operate an internal risk management system, for meeting, adequately, patient safety standards.

The major qualitative difference, between the two groups of organizations, stems from the different level of visibility they have in their operation and ultimately, from their different access to resources.

Both types of organizations need tools for monitoring their performance, beyond those they maintain for compliance with CMS, and their current multi-stack approach isn’t enough to ensure sustainable performance.

A few important observations from the findings of the two studies:

i) There are few efforts been made recently by research, in evaluation and development of MFWs for healthcare centres’ sustainable performance, mainly due to the complexity of the task.

ii) Even fewer are the MFWs that have been developed and validated, under rigorous analytical techniques, for ensuring that the monitored constructs are reliably estimated.

iii) From the applicable MFWs, tracking healthcare organizations’ performance, few are designed with operational resiliency and sustainability in mind.

iv) Most performance MFWs tend to adopt past and established practices, adjusted to the environment being monitored. There are few frameworks designed with scalability, automation or performance optimization in mind.

v) Most organizations tend to adopt multiple monitoring systems in practice, drawing methods from research, both for covering their own needs and for meeting compliance obligations. This design incurs significant costs annually, while even high-performing organizations are struggling to sustain the quality of their services.

vi) The distinctive features separating the two groups into high-performing and low performing organizations - such as adequate and reliable monitoring, performance optimization, investment in technology, increased automation - show that monitoring plays an integral role in organizations’ performance.

vii) The dimensions of the qualitative difference, between the two groups, in performance - adequate and reliable monitoring, performance optimization, investment in technology, increased automation etc. - align with the categories of the categorization framework, which further validates the framework’s capability to identify dimensions of monitoring that foster sustainable performance.

4. Discussion

CMS’s Monitoring Methodology - Final Remarks 

From the outcomes of the study, it is evident the primary role CMS’ monitoring and policy play in the sustained performance of the HCS, which is its primary mission.

Below, we briefly analyse CMS’s monitoring, from the lens of the categorization framework of the study.

The main monitoring frameworks it operates are under CoP [46], the Hospital Value-based Purchasing (HVBP) [49,53], the Hospital Readmission Reduction (HRR) [50], Hospital-Acquired Condition (HAC) [51], the Comprehensive Care for Joint Replacement (CCJR) [52] and the Hospital Overall Quality Star Rating (HOQSR) [54] Programs.

Through CoP, CMS determines the minimum operation safety and preparedness criteria, for healthcare centres to remain Medicare-Medicaid-eligible, which is essential for their financial viability, with the aim of building a baseline performance across the HCS. HVBP, HRR, HAC and CCJR quality standards act as incentivizers or penalizers of healthcare centres’ performance.

CMS’s monitoring methodology is a combined Measuring and Scoring approach, with built-in mechanisms for delivering valid performance evaluations [Suppl2].

Despite its robustness, there have been reported caveats that need to be rectified to operate a truly value-based care model.

As an example, while measurements of institutions’ performance are scientifically sound, when evaluated individually, their validity weakens, when bundled into single star ratings [65]. Pollock et al. [65], showed that due to CMS’s MFW sensitivity in the HOQSR program, a hospital’s rating can swing between a 3-star and a 5-star rating. In addition, Jianhui et al.l [66] highlight that care rated with 5 stars, due to high patient experience scores, can mask underperformance, with below-average mortality rates, across care paths that are outside HOQSR’s monitoring perimeter.

The processing of the submitted data to the Inpatient and Outpatient Quality Reporting Systems and effective decision making on payments takes place annually, reflecting performance prior to one year back, thus these ratings don’t convey an informative view of how an organization operates now.

Although CMS documents clearly the monitoring methodology it applies, the strong involvement of stakeholder/expert groups and manual procedures (e.g., audits, indicators definition, measure selection etc.) in the monitoring process [54], may set the creditworthiness of the process at stake.

The risk of failure against performance targets isn’t quantified and exact performance targets for the sustained operation of the HCS aren’t set [46] for each organization, hence CMS doesn’t cater, for organizations staying on performance track.

High-Performers’ Tracking of Progress - Final Remarks 

It comes out that, due to the highly complex cases reaching these healthcare centres, they need to invest significantly on the co-ordination of care, on research-embedded care pathways and hence on extensive quality measurement.

These requirements can’t be covered by regulatory bodies’ monitoring or funding mechanisms.

Besides, for the core of their monitoring system, their IQSs, it can be inferred, from their spend on IT – 3% on average of their annual opex [Suppl4,69] - and their estimated spend on Labor - 60% on average of their annual opex [Suppl4,69] -, that these systems are likely multi-stack, with some parts likely duplicated, not fully-automated, requiring human intervention for their operation. That structure of IQSs is expected, considering their composite model (§Results) and the lack of a unified approach in monitoring, and it accounts for an inflated annual cost.

Low-Performers’ Tracking of Progress - Final Remarks 

The lack of investment in IQSs and technology, due to the limited budget of these organizations, doesn’t allow them enough visibility into their weaknesses.

Maintaining compliance with CMS, through measures’ monitoring isn’t capable to help them exit the crisis, while the cost of regulatory compliance exacerbates their crisis. States complement CMS’s baseline performance monitoring, through state-specific measures’ monitoring, licensure statutes, inspections and other actions, but they cannot revert their route to closure of facilities/departments.

A Conceptual Framework for the Evaluation and Development of MFWs 

Based on the meaningful outcomes of the categorization framework, used in this study, and its discriminating capability of a broad sample of monitoring approaches, against their capability to promote organizations’ sustainability, a conceptual framework was created, for the aim of informing the evaluation and future development of MFWs, for healthcare centres and, potentially, for other components of the HCS, as well.

One instantiation of the conceptual framework and an example of its application in practice, has been the categorization framework of this study.

The principal design of the conceptual framework has been the adoption of a set of properties, any MFW is expected to have, for providing the required visibility to healthcare centres that will enable them to sustain their vision, while operating as part of a sustainable HCS. These are:

1. Operational - the monitoring process should not interfere with any healthcare process.

2. Reliable - it should provide correct view of the constructs it monitors. Changes in estimates, should reflect analogous changes in the constructs.

3. Feasible and Practical - it should not demand significant investment by an organization, and it should use existing technology efficiently.

4. Scalable and Adaptable - it should be able to operate reliably, under varying contexts.

5. Invariant - estimates of monitored constructs should not vary, because of any parameter change, beyond constructs.

6. Interpretable - it should be capable to explain any variation of the constructs’ estimation.

7. Transparent

8. It should track the risk of missing targets.

The reason ‘monitoring of risk’ is included in the properties, is, first, to provide a measure of the tracking progress of an organization and hence of its resiliency.

Secondly, to connect each MFW with the domain of finances, for the purpose of enabling payment and incentivization mechanisms over the MFW, linked to reliable and transparent performance aspects, and to financial instrument applications.

It becomes apparent that this property could directly and further propel organizations’ sustainability.

All properties are further validated, as promising dimensions of MFWs for organizations’ performance, by the 4Cs framework [27,28,29,30], as they rigorously act towards empowering the 4Cs, hence decreasing the gap from the performance frontier.

In Table 4 is shown, how the properties of the conceptual framework align with the categorization framework and the 4Cs framework.

5. Conclusions

This study highlights the strengths and the weaknesses of the recent monitoring approaches applied in practice, towards providing healthcare organizations with the required visibility, into their operation, for performing sustainably. Furthermore, it identifies the benefit of using a unified approach in evaluating and building MFWs, by instantiating the conceptual model that emerged from this study. With the properties of estimating performance reliably, being efficient and practical, scalable, invariant, interpretable, transparent and capturing risk of failure, the monitoring mechanisms produced not only enable organizations quantify their progress in sustainability but also help them improve their sustainability.