Paper‐and‐Pencil vs. Electronic Patient Records: Analyzing Time Efficiency, Personnel Requirements, and Usability Impacts on Healthcare Administration

Matthias Fabian Berger; Johanna Petritsch; Andrzej Hecker; Sabrina Pustak; Birgit Michelitsch; Chiara Banfi; Lars-Peter Kamolz; David B. Lumenta

doi:10.20944/preprints202409.1883.v1

Submitted:

23 September 2024

Posted:

24 September 2024

You are already at the latest version

Abstract

Objective The healthcare industry has witnessed a paradigm shift with the adoption of Electronic Patient Records (EPRs), transitioning from traditional Paper and Pencil (P&P) methods to sophisticated, technology-driven documentation systems. However, the integration of EPRs into clinical settings has introduced new challenges. This study investigates workflow consequences of switching from P&P to EPR systems. Methods In this study, two independent observers audited surgical ward rounds to assess the effects of transitioning from P&P to EPR. These audits captured number of medical personnel and five key aspects before and after EPR implementation. Additionally, the EPR system's usability was gathered through the System Usability Scale (SUS) and the Post Study System Usability Questionnaire (PSSUQ). Results A total of 192 observations using P&P and 160 with the EPR system were analyzed. Results indicated that physicians spent a lower proportion of time in patient rooms using the P&P modality (median = 0.14, IQR = [0.06, 0.24]) compared to the EPR system (median = 0.19, IQR = [0.12, 0.29]). Conversely, nurses spent more time with the P&P modality (median = 0.13, IQR = [0.08, 0.18]) than with the EPR system (median = 0.10, IQR = [0.06, 0.13]). The typical team in the P&P modality comprised two physicians and four nurses, while teams in the EPR system generally included two or three physicians and three nurses. Usability assessments yielded scores rated as "Not Acceptable." Conclusion The EPR is already known to contribute to an increased administrative burden. Our findings generally align with this view in the physician sample, whereas nurses did not encounter difficulties switching systems. However, usability scores indicated that the EPR system falls short of meeting user acceptance expectations. Given the continuous increase in administrative workload, our results suggest that ururcing administrative tasks could enhance the efficiency of medical duties performed at patient's bedside.

Keywords:

digital medicine

;

electronic patient record

;

electronic health record

;

clinical decision support

;

data science

Subject:

Medicine and Pharmacology - Clinical Medicine

Introduction

Electronic patient records (EPRs), also called electronic health records (EHRs), have become increasingly prevalent in the healthcare industry, revolutionizing the way patient information is recorded and stored. This shift from traditional Paper and Pencil (P&P) records to electronic health records has brought about numerous changes and advancements in healthcare documentation methods.

EPRs offer transformative advantages, playing an instrumental role in reshaping global healthcare environments. One central advantage is the creation of standardized and structured patient information databases, facilitating more efficient information retrieval and supporting robust clinical decision-making processes.[1] Complementing this is the recognized enhancement of patient care delivery efficiency, with professionals gaining timely and precise patient information access.[2] Within the healthcare sector, the significance of information technology (IT) in EPRs has been emphasized, notably in offering comprehensive information crucial for delivering high-quality care.[3,4] On a broader scale, EPRs aim to support nationwide health infrastructures by ensuring system interoperability, establishing a unified framework for addressing and understanding global health issues.[5] From a historical perspective, the emergence and evolution of patient management systems in response to growing data volumes and technological advancements suggest a trajectory where EPRs become integral to healthcare infrastructures.[6] Together, these perspectives highlight the pivotal role of EPRs in promoting more informed, efficient, and adaptable healthcare systems.

Electronic Patient Records on the other hand have increasingly come under scrutiny for their apparent misalignment with clinical workflows, and they have been identified as a salient factor contributing to physician burnout.[7] Clinicians often encounter difficulties in distinguishing between analogous patient data due to suboptimal interface designs. This results in disruptions to the workflow and cognitive overburdening as they grapple with vast volumes of data.[8] This trajectory in EPR development is not solely a reflection on senior clinicians, many of whom delegate their administrative burdens to residents (interns/junior medical staff) or if available to physician assistants (=PAs, who are less common or not available in public hospitals in Europe). This is concern directly reverted to EPR developers who, being predominantly fixed on function-based and list-centric designs, tend to being blinded or informed about intricacies of clinical workflows.[9] From the perspective of a majority of health care providers, there is usually no clinician engagement during the software development process. When involvement does occur, it frequently arises during the implementation phase, a later stage that requires significant time and elicits limited interest from clinicians outside their primary area of expertise[10].

This study aimed to evaluate the effects of transitioning from a P&P to an EPR system for medication and documentation during ward rounds in a major hospital’s surgical division in a public national European Union health care system (Austria), where PA’s are usually not available and the documentation requires to be performed by medical professionals. Specifically, the research focused on how this transition influenced the ward-based workflow (i.e., impact of administrative workload and personnel requirements) for clinical staff using planned two-week observational audits during weekday patient rounds in a surgical division.

Methods

This prospective observational study was approved by the institutional ethical board of the Medical University of Graz (vote #: 34-208 ex21/22). It was conducted at the Division of Plastic, Aesthetic and Reconstructive Surgery, Department of Surgery at the Medical University of Graz between November 2021 and May 2022.

We investigated to which extent the EPR use affected the time clinical personnel spend on hands-on patient care during surgical ward rounds and adopted two distinct workflows (Figure 1) to evaluate clinical routines before and after the introduction of the EPR system.

Workflow A: Observational Audit during Clinical Ward Rounds

Throughout the course of routine clinical ward rounds, two independent observers diligently executed two distinct, continuous audits, each extending over a period of two work weeks. The initial audit was conducted during a phase where Paper & Pen (P&P) constituted the dominant method of documentation. This was contrasted with a subsequent audit, undertaken three months following the transition to the Electronic Patient Record (EPR) system. The system utilized in this study was the OpenMEDOCS hospital information system (HIS), a comprehensive platform designed to facilitate the documentation, retrieval, and analysis of patient data in a clinical setting.[11] The total duration of each visit was calculated as the cumulative time (in seconds) encompassing five key components: 1) preparatory time prior to entering the patient’s room, 2) the duration of the physicians’ documentation time in the patient’s room, 3) the time allocated by nurses for documentation in the patient’s room, 4) the extent of direct patient interaction by the physicians, and 5) the time invested in applying bandages. Instances where the duration was recorded as 0 seconds were treated as missing data. The objective was to conduct a comparative analysis of these five parameters between the P&P and EPR documentation modalities. To mitigate baseline disparities arising from variations in total visit duration, proportions were calculated by dividing the time dedicated to each of the five (1-5) aspects by the overall ward round time. Furthermore, the number of medical personnel present during the ward round was collected. The data for the Paper & Pencil entry mode were collated between August 30, 2021, and September 10, 2021, whereas the data pertaining to the electronic mode were gathered between March 7, 2022, and March 18, 2022.

Statistical Analysis for Workflow A

Continuous variables deviated from normal distribution and were descriptively summarized using median and interquartile range [IQR]. Categorical variables were presented as absolute and relative frequencies. Note that the number of medical personnel was treated as a categorical variable to facilitate the comparison between the different modalities. Between-subject comparisons of continuous variables were performed with Wilcoxon rank sum test. Associations among categorical variables were tested by means of Fisher’s exact test. Visit durations for the same patient on consecutive days might have been influenced by medical staff’s prior familiarity with the patient. To mitigate this potential bias, sensitivity analyses were performed where only the first patient visits were included. P-values < 0.05 were deemed statistically significant. The analysis was executed using R software (version 4.2.2)[12].

Workflow B: Post-Implementation EPR Usability Survey

Separately from the observational audits, we polled the clinical staff (nurses, residents and attendings) three months after the EPR’s introduction. We evaluated the EPR usability as perceived using the firsthand experiences by two established instruments served as poststudy questionnaires: the System Usability Scale (SUS) and the Post Study System Usability Questionnaire (PSSUQ).[13] The SUS is widely used and accounts for 43% of post-study questionnaires in the literature.[14] The 10-question questionnaire is based on a 5-point Likert scale (1 = strongly disagree; 5 = strongly agree). The calculated combination yields an overall usability score ranging from 0–100%.[15] A modified score ranking by Bangor et al. (Figure 2) was used to interpret the overall usefulness of each patient.[16] The PSSUQ-3 is conceptualized to evaluate users’ subjective satisfaction with computerized systems and applications. The 16-question questionnaire is based on a 7-point Likert scale (1 = strongly agree; 7 = strongly disagree). The score is divided into an overall average (derived from the mean of all 16 components). This encompasses a ‘system usefulness’ subscale, gauging the system’s intuitiveness and learnability (average of items 1–6); an ‘information quality’ subscale, reflecting the feedback relayed to the user by the system (average of items 7–12); and an ‘interface quality’ subscale, evaluating the use’s affinity for the system and its alignment with anticipated functionalities (average of items 13–16). Normally, a lower score on the PSSUQ correlates with a higher perceived usability of the system.[17] For the purpose of enhanced readability, the PSSUQ scores have been inverted, that an increased score is directly associated with improved system usability. Both questionnaires, the SUS and PSSUQ, are available in the supplementary materials (Table S3, Table S4).

Statistical Analysis for Workflow B

The statistical evaluation entailed determining the means and standard deviations (SD) for continuous variables, alongside the computation of frequencies and proportional frequencies for categorical variables. In cases where data did not adhere to a normal distribution, the median and the interquartile range (IQR) were employed.

Results

Duration of Administrative Work on Patient Care During Surgical Ward Rounds

We collected 192 observations utilizing the P&P system and 160 observations employing the EPR system. No significant differences were observed between the P&P system and the EPR system in the overall duration of ward rounds and in three out of five critical components: the preparatory time before entering the patient’s room, the extent of direct patient interaction by the physicians, and the time spent applying bandages (p ≥ 0.470). However, significant differences between data entry modalities were observed in the proportion of time spent inside the patient’s room by physicians (p < 0.001) and nurses (p < 0.001). Physicians spent less time inside the patients room using the P&P modality (median=0.14, IQR=[0.06,0.24]) as compared to the EPR system (median=0.19, IQR=[0.12,0.29]). In contrast, the proportion of time spent by nurses inside the patients room was higher with the P&P modality (median=0.13, IQR=[0.08, 0.18]) as compared to the EPR system (median=0.10, IQR=[0.06,0.13]). Note that these significant differences were not replicated in our sensitivity analysis including only first visits (see Table S2)

Medical Personnel Distribution

The proportion of physicians attending to patients differed between the two data entry modalities (p < 0.001). As shown in Figure 3, the presence of two physicians was the most common occurrence in the P&P modality (75%), whereas in the EPR system two (45%) or three (36%) physicians were most common.

The proportion of nurses attending to patients differed between the two data entry modalities (p = 0.003). In the P&P modality, the presence of four nurses was the most common occurrence (50%), whereas in the EPR system three nurses (43%) were most common. See Figure 3, right plot.

Usability by Professional Group

Among the participants, the response rate for registered nurses was 53% (9 out of 17), for residents it was 88% (7 out of 8), and for attending physicians it was 50% (7 out of 14).

The System Usability Scale (SUS) (Figure 4) revealed an overall mean score of 37.6 (SD = 19.9). When disaggregated by professional designation, nurses reported an average score of 40.8 (SD = 14.8), residents had a slightly higher mean at 46.8 (SD = 21.8), while attending physicians yielded the lowest mean score of 24.3 (SD = 19.0). In interpreting the findings presented by Bangor et al.[16], it is concluded that all scores are rated as “Not Acceptable”. (The differences were statistically not significant (p=0.085, Kruskal wallis).

In the Post-Study System Usability Questionnaire (PSSUQ) (Table 1) the overall mean score was 3.4 (SD = 1.1). Nurses reported a mean score of 3.3 (SD = 1.2), residents had 4.0 (SD = 1.2), and attending physicians averaged at 3.0 (SD = 0.9). Again, the Kruskal-Wallis test rendered a p-value of 0.304, indicating no significant differences across the groups.

A deeper examination of the PSSUQ sub-domains further elaborated on these trends. For the SYSUSE segment, the mean score across all respondents was 3.5 (SD = 1.3), with a Kruskal-Wallis p-value of 0.243. The INFOQUAL sub-domain resulted in an overall mean of 3.3 (SD = 1.1) and a p-value of 0.315. Lastly, for the INTERQUAL sub-domain, participants reported a mean score of 3.4 (SD = 1.3), and the p-value stood at 0.717. (See Figure S5 for detailed chart view).

Discussion

The aim of this study was to investigate the change in administrative workload on the time spent for patient care and change in number of personnel requirements following the implementation of an Electronic Patient Record (EPR) system, and to assess the systems’ usability and user acceptance across the involved professional groups.

Administrative Workload Time

The distribution of time during ward rounds has the potential to affect the concentration levels of healthcare professionals and the frequency of medical errors.[18] In utilizing the EPR system, physicians demonstrated a significant higher workload time on using the EPR than it was by using P&P. As from a proportional standpoint that means, that physicians spend more time on documentation, than spending time with the patients themselves. Nurses showed the exact opposite in our study. Spending less time on administrative work means more time for addressing the patient from a nursing point of view. With a more detailed examination of ward round duties in both the physician and nursing groups, our results align with a similar study from 2008 found in the literature.[19]

In modern healthcare settings, medical professionals are increasingly expected to possess administrative skills in addition to their clinical expertise. A significant amount of a physician’s time is dedicated to administrative tasks, which can detract from direct patient care. This shift towards more administrative responsibilities is linked to lower job satisfaction among doctors.[20] The adoption of the EPR has been identified as a factor that contributes to this increased administrative load. Ammenwerth and Spötl’s study shows, that medical professionals spend almost as much time on documentation as they do on taking care of patients directly.[21] Research by Woolhandler and Himmelstein indicates that about one-sixth of a physician’s working hours are spent on administrative work unrelated to patient care.[22] This substantial investment of time in administrative duties can reduce the opportunities for patient interaction, potentially impacting the quality of healthcare services provided. Capturing the spirit of times, the study by Liu et al. demonstrates that the use of an AI-powered clinical documentation tool improved efficiency for many clinicians, reducing time spent on electronic health records and alleviating frustration.[23] Although the benefits and improvements were not universally experienced by all participants, we believe that AI-supported administrative tasks will gain significant momentum in the near future.

Number of Medical Personnel Requirements

Both physicians and nurses demonstrated notable differences in their staffing needs depending on the system used, highlighting that the demand for personnel is dependent on the chosen documentation method. The analysis revealed that a higher number of personnel was required to operate the EPR among physicians compared to P&P. Conversely, nurses demonstrated an opposite trend, whereby fewer nurses were needed when using the EPR as opposed to P&P.

In a study, Watson et al. compared residents with nurses and physician assistants (PA), with the latter showing a higher workload than physicians.[24] Yet, only a few countries have integrated PAs into their workforce[25], this potentially ensures delegation of administrative tasks into trained supportive hands from a medical perspective. Since the early 1960s, with the introduction of a new health care provider model, the first PA-training-program was developed to assist physicians to such an extent that it increased the capacity of available doctors and nurses.[26,27] The integration of PAs into hospitals across European countries has demonstrated significant alleviation of workload pressures and fostered positive interactions between physicians and PAs.[28,29] With increasing use of EPR, the integration of an administrative skillset into the curriculum for physician assistants should occur early in their education, enabling organizations to support role integration and enhance the ability to facilitate sustainable healthcare delivery.[29,30] The introduction of PAs thus demonstrates the potential not only to conserve medical personnel by reducing the number of physicians required, but also to decrease the time spent on administrative tasks. This can consequently streamline the efficiency of processes that currently burdens physicians.

Usability

There is no statistically significant difference in both the System Usability Scale (SUS) and Post-Study System Usability Questionnaire (PSSUQ) scores among the professional groups.

On examining the SUS scores, while the mean scores differed among nurses, residents, and attending physicians, with attending physicians presenting the lowest usability score, these variations were not found to be statistically significant (p=0.085). According to the benchmark scale by Bangor et al.[16], the overall usability yielded an unsatisfactory result. Considering this, it is deemed as “Not Acceptable”.

Similar results were found when interpreting the PSSUQ. PSSUQ scores across professional groups indicate room for usability improvements. Nursing staff rated system usability at a mean score of 3.3, while residents had a more favorable 4.0 mean score. Attending physicians were least satisfied, registering a mean score of 3.0 with low standard deviation, suggesting a homogeneous yet less favorable opinion. The combined mean score for all groups was 3.4, slightly above average, but still suggesting the need for enhancements. Further scrutiny into specific categories such as system usability (SYSUSE), information quality (INFOQUAL), and interface quality (INTERQUAL) revealed similar trends. Notably, residents gave the highest scores in the SYSUSE category with a mean of 4.3, whereas attending physicians generally provided lower scores across most categories.

These results align with usability challenges of the EPR system in the literature.[31,32,33] Also, Kaipio et al. mentioned the different usability aspects throughout professional groups, especially physicians and nurses.[34] Common issues identified include data overload, missing/hidden information, difficulty identifying trends, and the need for manual calculations. Another problem is the poor visibility and lack of distinctiveness in finding required information on-screen, leading to difficulty in generating in-patient selection.[8,35] Usability violations in EPR systems can disrupt physician workflow processes, increase documentation time, contribute to clinician frustration, burn out and potential patient safety issues.[36,37,38] In our study, the most common free-text replies and verbal feedback corresponded with existing literature. Issues such as slow system responsiveness were observed as data mining accumulated from various sources, leading to delayed data loading. Additionally, there was a lack of an overview of relevant data, an excessive number of options/buttons for simple tasks, and a high number of clicks required to perform these tasks.

In a hospital system that has not been optimized for workflow efficiency, the utilization of clinical personnel for administrative tasks emerges as a significant economic inefficiency. This issue is compounded by the lack of standardized workflows across, and even within, individual hospital systems, leading to habitual inconsistencies that further convolute clinical procedures. Such complexities manifest in various ways, from the excessive number of clicks required to execute specific tasks, to the lack of intuitive interface design, necessitating convoluted workarounds. Furthermore, existing enterprise resource planning systems, such as SAP (Systems, Applications, and Products in Data Processing), often do not align seamlessly with the unique demands of clinical workflows. While customization of these platforms is possible, it often incurs substantial follow-up investment costs and typically involves limited input from clinicians. This creates a catch-22 situation, where the very resources needed for optimization are instead contributing to existing inefficiencies. In addition, the potential benefits of economies of scale in the software industry have not been fully utilized in clinical workflows, resulting in missed chances to improve efficiency and cost-effectiveness.

Limitations

Our observations were confined to a single division and were based on a clinical routine scenario, which demonstrated variability among clinical staff members, with the resident being the only consistent presence. Also, the research did not encompass an objective evaluation of in-patient interactions involving varying complexities of patient histories with EPR usability.

Conclusions

The EPR is already known as a factor contributing to an increased administrative burden. Our findings are generally consistent with this view in the physicians’ sample, whereas nurses did not encountered difficulties switching systems. However, usability scores indicated that the EPR system falls short of user acceptance expectations, with all professional categories deeming the system “Not Acceptable”. This study indicates that there may be potential advantages to a new electronic system for data entry, but there are explicit usability flaws that prevent its acceptance. Yet, limited adopted in European countries, the incorporation of Physician Assistants (PAs) was shown to significantly ease workload pressures and encourage positive interactions between medical staff and PAs. Given the continuous increase in administrative workload, our results suggest that outsourcing administrative tasks can enhance the efficiency of medical duties performed at patient’s bedside.

Supplementary Materials

The following supporting information can be downloaded at the website of this paper posted on Preprints.org.

Author Contributions

DBL, MFB, and BM were responsible for the conceptualization of the study. MFB and DBL also contributed to the funding acquisition and project administration, including protocol development and gaining ethical approval. JP and SP handled the investigation, including patient recruitment, data curation, and execution of the study methods. CB and AH performed the formal data analysis. MFB prepared the original draft of the manuscript. All authors contributed to the review and editing of the manuscript, approving the final version for submission.

Funding

This study was supported by the Medical University of Graz.

Ethical approval

This study was approved by the institutional ethical board of the Medical University of Graz (vote #: 34-208 ex21/22).

Informed Consent

Not applicable

Data Availability Statement

The data presented in this study are available upon request from the corresponding author.

Acknowledgment

We would like to thank all those who have actively collaborated in this study.

Conflict of interest

All authors declare that they have no conflicts of interest.

Guarantor

David Benjamin Lumenta (DBL)

References

Shahmoradi, L.; Darrudi, A.; Arji, G.; Nejad, A.F. Electronic Health Record Implementation: A SWOT Analysis. Acta Med. Iran. 2017, 642–649.
Phillips, M.R. The Electronic Health Record Will Be What We Make It. Ann. Surg. 2020, 272, 229. [CrossRef]
Haas, P.G. Grundlagen, Anwendungen, Potenziale; Springer, 2006;
Haux, R. Strategic Information Management in Hospitals: An Introduction to Hospital Information Systems; Springer Science & Business Media, 2004; ISBN 978-0-387-40356-4.
Fragidis, L.L.; Chatzoglou, P.D. Implementation of a nationwide electronic health record (EHR): The international experience in 13 countries. Int. J. Health Care Qual. Assur. 2018, 31, 116–130. [CrossRef]
Hammond, W.; Bent, B.; West, V.L. Goodbye Electronic Health Record? In Digital Professionalism in Health and Care: Developing the Workforce, Building the Future; IOS Press, 2022; pp. 107–111.
Melnick, E.R.; Fong, A.; Nath, B.; Williams, B.; Ratwani, R.M.; Goldstein, R.; O’Connell, R.T.; Sinsky, C.A.; Marchalik, D.; Mete, M. Analysis of Electronic Health Record Use and Clinical Productivity and Their Association With Physician Turnover. JAMA Netw. Open 2021, 4, e2128790. [CrossRef]
Shin, G.W.; Lee, Y.; Park, T.; Cho, I.; Yun, M.H.; Bahn, S.; Lee, J.-H. Investigation of usability problems of electronic medical record systems in the emergency department. Work 2022, 72, 221–238. [CrossRef]
Janett, R.S.; Yeracaris, P.P. Electronic Medical Records in the American Health System: challenges and lessons learned. Ciênc. Saúde Coletiva 2020, 25, 1293–1304. [CrossRef]
Høstgaard, A.M.; Bertelsen, P.; Nøhr, C. How are clinicians involved in EHR planning? A process analysis case study of a region in Denmark. In MEDINFO 2010; Ios Press, 2010; pp. 121–125.
ANDRIEU, J.; LEIKAUF, H.; MEINX, E.; KÖNIG, K. Krankenhausinformationssystem openMEDOCS; Landesrechnungshof Steiermark, 2003; p. 130;
Available online: https://www.r-project.org/ (accessed on Aug 8, 2023).
Sauro, J.; Lewis, J.R. Quantifying the user experience: Practical statistics for user research; 1st ed.; Morgan Kaufmann, 2012; ISBN 978-0-12-384968-7.
Lewis, J.R. The system usability scale: past, present, and future. Int. J. Human–Computer Interact. 2018, 34, 577–590. [CrossRef]
Brooke, J. SUS-A quick and dirty usability scale. Usability Eval. Ind. 1996, 189, 4–7.
Bangor, A.; Kortum, P.; Miller, J. Determining what individual SUS scores mean: Adding an adjective rating scale. J. Usability Stud. 2009, 4, 114–123.
Sauro, J.; Lewis, J.R. Quantifying the user experience: Practical statistics for user research; 2nd ed.; Morgan Kaufmann, 2016; ISBN 978-0-12-384968-7.
Launer, J. What’s wrong with ward rounds? Postgrad. Med. J. 2013, 89, 733–734. [CrossRef]
Asaro, P.V.; Boxerman, S.B. Effects of Computerized Provider Order Entry and Nursing Documentation on Workflow. Acad. Emerg. Med. 2008, 15, 908–915. [CrossRef]
Mechanic, D. Physician discontent: challenges and opportunities. Jama 2003, 290, 941–946.
Ammenwerth, E.; Spötl, H.-P. The time needed for clinical documentation versus direct patient care. Methods Inf. Med. 2009, 48, 84–91. [CrossRef]
Woolhandler, S.; Himmelstein, D.U. Administrative work consumes one-sixth of US physicians’ working hours and lowers their career satisfaction. Int. J. Health Serv. 2014, 44, 635–642. [CrossRef]
Liu, T.-L.; Hetherington, T.C.; Stephens, C.; McWilliams, A.; Dharod, A.; Carroll, T.; Cleveland, J.A. AI-Powered Clinical Documentation and Clinicians’ Electronic Health Record Experience: A Nonrandomized Clinical Trial. JAMA Netw. Open 2024, 7, e2432460. [CrossRef]
Watson, M.D.; Elhage, S.A.; Scully, C.; Peterson, S.; Gulledge, M.; Cunningham, K.; Sachdev, G. Electronic health record usage among nurse practitioners, physician assistants, and junior residents. J. Am. Assoc. Nurse Pract. 2020, 33, 200–204. [CrossRef]
Maier, C.B.; Batenburg, R.; Birch, S.; Zander, B.; Elliott, R.; Busse, R. Health workforce planning: which countries include nurse practitioners and physician assistants and to what effect? Health Policy 2018, 122, 1085–1092. [CrossRef]
Hudson, C.L. Expansion of medical professional services with nonprofessional personnel. JAMA 1961, 176, 839–841. [CrossRef]
Stead, E.A. Conserving costly talents--providing physicians’ new assistants. JAMA 1966, 198, 1108–1109. [CrossRef]
Meyer-Treschan, T.; Stegemann, A.K.; Sebastian, J.; Hatwich, S.; Beiderlinden, M.; Siepe, R.B.; Veltjens, B.; Farhan, N.; Siegmüller, J. Gesundheitsversorgung in Deutschland durch Mitarbeit von Physician Assistants im ärztlichen Team. Gesundheitswesen 2023, 85, 181–187. [CrossRef]
Roberts, S.; Howarth, S.; Millott, H.; Stroud, L. ‘What can you do then?’ Integrating new roles into healthcare teams: Regional experience with physician associates. Future Healthc. J. 2019, 6, 61. [CrossRef]
Barnett, J.S. Incorporating Electronic Medical Records into the Physician Assistant Educational Curriculum: J. Physician Assist. Educ. 2013, 24, 48–54. [CrossRef]
Bloom, B.M.; Pott, J.; Thomas, S.; Gaunt, D.R.; Hughes, T.C. Usability of electronic health record systems in UK EDs. Emerg. Med. J. 2021, 38, 410–415. [CrossRef]
Howe, J.L.; Adams, K.T.; Hettinger, A.Z.; Ratwani, R.M. Electronic Health Record Usability Issues and Potential Contribution to Patient Harm. JAMA 2018, 319, 1276–1278. [CrossRef]
Tyllinen, M.; Kaipio, J.; Lääveri, T. Usability Analysis of Contending Electronic Health Record Systems. In Proceedings of the ITCH; 2019; pp. 430–435. [CrossRef]
Kaipio, J.; Kuusisto, A.; Hyppönen, H.; Heponiemi, T.; Lääveri, T. Physicians’ and nurses’ experiences on EHR usability: Comparison between the professional groups by employment sector and system brand. Int. J. Med. Inf. 2020, 134, 104018. [CrossRef]
Alami, J.; Borowitz, S.; Riggs, S.L. Usability Challenges with EHRs During Pre-Rounding in the Pediatric Acute Care Department. Proc. Hum. Factors Ergon. Soc. Annu. Meet. 2020, 64, 1282–1286. [CrossRef]
Berg, G.M.; Shupsky, T.; Morales, K. Resident Indentified Violations of Usability Heuristic Principles in Local Electronic Health Records. Kans. J. Med. 2020, 13, 84–89.
Melnick, E.R.; Dyrbye, L.N.; Sinsky, C.A.; Trockel, M.; West, C.P.; Nedelec, L.; Tutty, M.A.; Shanafelt, T. The Association Between Perceived Electronic Health Record Usability and Professional Burnout Among US Physicians. Mayo Clin. Proc. 2020, 95, 476–487. [CrossRef]
National Academies of Sciences, E. Taking action against clinician burnout: a systems approach to professional well-being. 2019.

Figure 1. Clinical Workflow Assessment.

Figure 2. Modified SUS Ranking by Bangor et al.

Figure 3. Overall Distribution of Observed Physicians (left plot) and Nurses (right plot).

Figure 4. SUS Scores Across Professions.

Table 1. PSSUQ Scores Across Professions.

PSSUQ index*	Nurse Mean, (SD) [%]	Resident Mean, (SD) %	Attending Mean, (SD) %	Overall Mean, (SD) %	p-value
Overall	3.3 (1.2) [38.33]	4.0 (1.2) [50.00]	3.0 (0.9) [33.33]	3.4 (1.1) [40.00]	0.304
System Usability (SYSUSE)	3.3 (1.5) [38.33]	4.3 (1.3) [55.00]	3.0 (0.9) [33.33]	3.5 (1.3) [41.67]	0.243
Quality of the information (INFOQUAL)	3.1 (1.2) [35.00]	3.8 (1.0) [46.67]	3.0 (0.9) [33.33]	3.3 (1.1) [38.33]	0.315
Quality of the interface (INTERQUAL)	3.5 (1.4) [41.67]	3.7 (1.6) [45.00]	3.1 (1.0) [35.00]	3.4 (1.3) [40.00]	0.717
*Rating score from 1 = worst [0%] to 7 = best [100%].

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

Copyright: This open access article is published under a Creative Commons CC BY 4.0 license, which permit the free download, distribution, and reuse, provided that the author and preprint are cited in any reuse.