Planning Audit and Feedback Interventions in Health Care Organisations. An Account from an Italian National Program for Audit & Feedback Implementation

1. Introduction

Among interventions aimed at changing health providers’ behaviors, “Audit & feedback” (A&F) has been emerging as one of the most promising as its effectiveness has been shown in different settings. In an A&F process, an individual’s professional practice or performance is measured and then compared to professional standards or targets [1,2].

Indeed, providing health professionals with structured reports on their performance can lead to quality improvement when process or outcomes indicators highlight potential problems concerning the delivery of specific health care interventions or the clinical outcomes achieved. However, many factors influence the yield of A&F interventions, including the context in which they were applied, the type of targeted behaviors, and how they are structured and delivered [1,3,4].

A&F includes a wide range of interventions which differ in terms of type of quality indicators used, structure and frequency of the report feedback to health professionals. While it is known that A&F interventions can improve clinical practice in general, it is much less clear how they should be applied in health care organizations. Recommendations on how an A&F system should be conceived and developed to be able to fully exploit its potential have been provided by experts in the field [4,5,6]. However, the actual applicability of these recommendations - based on the available evidence, mostly based on UK and North America experiences, as well as on assumptions from theories of behavior change and common sense - should not be taken for granted. A&F are indeed complex interventions strongly influenced by the context, whose implementation poses several challenges in terms of competences and skills to be involved in its conceptualization and design, and technical problems to be considered.

In Italy, the Ministry of Health launched in 2019 a national research program (NET-2016-02364191) [7,8], aimed at exploring the worth of A&F interventions, with participating Institutions in seven regions (Lazio, Friuli-Venezia Giulia, Piedmont, Emilia-Romagna, Lombardy, Calabria, Sicily) conducting projects applying A&F initiatives in different settings (Table 1).

Aim of this paper is to outline how A&F interventions under the EASY NET project were designed at an early stage, to explore the extent to which current recommendations on desirable characteristics of an "ideal" A&F procedure were adopted.

2. Materials and Method

In the framework of the EASY-NET national research program a template was developed with the aim of collecting information on how the different A&F interventions were designed in order to meet the specific needs of each project.

The template was divided into six sections addressing the following components of the A&F intervention to be developed:

• description of the working group composition (i.e., competencies, skills and clinical and organizational responsibilities represented in the team in charge of designing the A&F intervention);
• targeted clinical behavior(s) (i.e., the clinical behavior(s) to be changed through the A&F intervention);
• selected indicators and their informative sources;
• the feedback procedures to be adopted (i.e., timing and frequency of the reports, as well as their structure);
• actions (if any) expected from the targeted health professionals (i.e., what health professionals were supposed to do or act on after feedback delivery);
• other intervention(s) if any, to be carried out along with the A&F procedure to sustain / reinforce its impact.

Information gathered through the template was then classified into four main topics (nature of the desired action - i.e. the targeted behavior-, type of data available for feedback, feedback display and feedback delivery), in line with the categorization used by Brehaut et al. 2016 for their recommendations [4].

3. Results

3.1. Nature of the desired action

In three of the nine projects (WP3A, WP3B, WP6) the A&F procedure was aimed at changing a targeted and clearly identifiable clinical behavior. In six circumstances (WP1A, WP1B, WP2, WP4, WP5, WP7) A&F was generically aimed at stimulating the attention to the quality of care provided to specific categories of patients. Moreover, the type of actions that intervention research teams expected from the recipients as consequences of the A&F procedure seems to be generic, being reported as “dialogue among colleagues” or “promotion of comparison among peers”. All the projects identified clinicians (GPs and/or specialists) as the recipients of the information. Managers and other professionals with organizational responsibilities were explicitly considered in almost all projects (except for WP4, WP5, WP6).

3.2. Nature of the data available for feedback

Indicators were planned to be developed in all projects from administrative databases. Moreover, in some cases ad hoc data collection was planned, using dedicated web-based databases in four cancer audits, self-completed questionnaires in one study, and interviews in two studies. Overall, the mean number of process/outcomes indicators considered in the A&F projects was 27 range (9 – 61). Most of them (75%) were process indicators, such as measures of adherence to guideline recommendations on therapies and clinical examinations. Indicators designed to describe the volume of activity (i.e. the number of patients/interventions included in the pathway/study, hospital ward or ED patient stay and their characteristics) were included in all the studies. Outcome measures (i.e., hospitalizations, length of hospital stay, mortality and complications rates) were relatively less considered, accounting for 25% of the whole number of indicators used.

The reference used as comparators were derived from recipient performance (changes over time), formal guidance or a peer group (mean performance of similar persons or organizations). In all the projects reference standards were drawn from scientific literature (i.e. international standards), from locally available data (average regional data, comparison with professionals of the same Primary Care Department) or from national law.

In 4 projects (WP1A, B, WP4, WP7) the provision of feedback was scheduled half yearly, but other frequencies were reported: annually (WP2, WP3B), every 3 months (WP3B), on demand (WP3A). One project did not indicate the feedback frequency.

The reference time varied from 6 months (WP1A, WP6 projects), 12 months (WP1B, WP7 projects) in the collected projects. In the other projects the reference time was not explicitly reported.

3.3. Feedback display and delivery

All the feedback strategies, according to the different frequency and timelines chosen for each project, allowed access to aggregated data that can be displayed through graphs and tables. These data were planned to be compared with reference standards reported in scientific literature or between different areas. Different feedback sending options were chosen: web platforms with access credentials (WP1, WP3 A), e-mails (WP6, WP7), workshops (WP2) and individual meetings (WP3B). While in three initiatives the use of economic incentives to encourage clinicians’ participation was mentioned, no additional intervention along with the A&F procedure was planned in the other regional projects. Main intervention features have been reported in Table 2.

4. Discussion

In this paper, we describe a wide experimental introduction of A&F in healthcare organizations in Italy, promoted by a national program specifically aimed at spreading these quality improvement strategies and assessing their impact on quality of care. The information collected at the beginning of the project through the survey carried out on the nine regional WPs (projects) gave us the opportunity to explore how these types of intervention were designed when applied in the context of health care organizations.

Three main relevant issues emerged from the original design of the A&F interventions developed by the regional projects.

The first issue concerns the specific aim of the intervention itself. According to Brehaut, intervention’s research teams should recommend actions that were consistent with established goals and priorities. Therefore, these should be explicit, specific, time-bound, recipient-defined, challenging but also attainable, with room for improvement and over which the recipient has control. Moreover, specific rather than general actions should be recommended to be more effective. Of course, compliance with these recommendations implies clear identification of a clinical behavior to be targeted and hopefully changed through the A&F intervention.

In 7 out of 9 analyzed projects the objectives of the A&F intervention were quite broad, with multiple purposes, aimed at promoting greater attention to the improvement of the quality of care in the clinical area considered. It was assumed that A&F, providing a broad description of the quality of care through a set of indicators, could have a general positive impact offering opportunity for discussion to the multidisciplinary clinical communities involved in the provision of care. In such a context, A&F seems implicitly seen more as a “clue” keeping together through the information provided the different stakeholders, than a tool aimed at changing few, well specified behaviors.

The second issue is about process and outcome indicators. The high number of process and outcomes indicators used in the regional projects described seems to be consistent with this broad, unfocused approach. If the goal is to provide information on the whole patterns of care in a clinical area, rather than changing a targeted behavior, several indicators are required. Of course, this raises several issues concerning the availability of timely and valid data and the extent to which recipients of reports based upon such a high number of indicators could be overwhelmed, rather than guided, by the amount of information received.

As far as indicators are concerned, it is worth noting the almost exclusive reliance on administrative databases. It is well known that administrative database is indeed a valuable tool when it comes at assessing quality of care, however there are relevant dimensions of quality that cannot be fully explored using only that source of information. Assessment of the appropriateness of use of health care interventions requires detailed clinical information on individual patients’ characteristics typically missing in administrative databases. The exclusive reliance on administrative databases also limits the timeliness and frequency of the feedback, being the reporting system conditioned on the actual availability of the data.

Overall, such unbalance between administrative and clinical databases is at odd with the extensive penetration of ITC technologies in health care organizations. Electronic medical records and laboratory databases, which are available in most health care settings, seem to be tools designed and used mainly for the management of single patients or for administrative management. Their potentials are still far from being fully exploited for quality assessment and improvement purposes. Electronic medical records are indeed much richer than administrative databases of detailed clinical information. That information can be made available to recipients much more frequently and timely. The relative underrepresentation in our sample of ICT experts in the teams responsible for the design of the A&F interventions points out the extent to which this aspect seems to be still overlooked.

The third issue concerns the feedback report strategy. In an A&F intervention, the research teams should keep in mind that feedback is more effective if a summary message and a visual display are both included and linked conceptually and visually. Moreover, providing feedback in more than one way (for example, combining spoken words and picture) and minimizing extraneous cognitive load (without overly complex information) for feedback recipients can positively affect the efficacy of the intervention.

When deciding on how to deliver a feedback intervention, according to the 15 suggestions, the A&F research teams should consider different actions [4]. Among these, actions worth considering are: addressing barriers to using feedback to reach the intended target audience; providing short, operational messages with optional information available for interested recipients; addressing credibility of the information, such as with the help of supervisor or colleague; preventing defensive reactions to feedback; and building feedback through social interaction, rather than passively receiving it. Little correspondence of these suggestions was found, at least in the planning phase of the projects analyzed.

Our account of how health care organizations in Italy approach the challenge of designing and implementing audit and feedback interventions obviously has limitations. Firstly, we provide a description on how A&F interventions have been conceived and planned at a very early stage. It is reasonable to assume that some changes in their structure may have occurred during the actual implementation phase, as some preliminary findings seem to show [9,10,11,12]. In addition, these projects are still ongoing, and therefore we do not have yet information on their actual impact on clinical practice, but many improvements have been possible, including through collaboration with the A&F International Meta lab, established under EASYNET. Furthermore, while health care organizations from seven regions in Northern, Central and Southern Italy were involved in our survey and the results should be considered representative of the Italian context, the extent to which they can be actually considered generalizable to other countries is questionable.

5. Conclusions

From the description of the projects designed in the different regional settings involved in the Italian National A&F Program EASY-NET, it can be concluded that these interventions, at least at an early stage, are mostly intended as "generic reminders" to generally improve the quality of care, rather than interventions aimed at changing specific clinical behaviors as described by the prevailing research literature and expert recommendations. The changes introduced during the course of the projects (some forced due to the COVID pandemic) and the final results obtained will offer interesting insights into the ultimate effectiveness of the different A&F strategies.