Multi-Phase, Multi-Method Usability Evaluation of an Enhanced Dosimetry Quality Assurance Checklist in Radiation Oncology: From Think-Aloud Testing to Near-Live Clinical Implementation

Most patient safety events in radiation therapy originate in treatment planning, and quality assurance (QA) checklists are widely deployed to mitigate them; however, these tools are rarely subjected to formal human factors evaluation before clinical release. Building on our prior work demonstrating suboptimal usability of an institutional dosimetry QA checklist (DQC) and its participatory, theory-driven redesign, this study evaluated an enhanced DQC using Borycki and Kushniruk’s multi-phase, multi-method usability evaluation framework, which integrates cognitive and socio-technical perspectives to create a “safety net” against usability problems and technology-induced errors. Three sequential phases were conducted at an academic medical center: (1) rapid think-aloud usability testing with two cohorts of dosimetrists and physicists (n = 10) separated by an improvement cycle; (2) remote simulation-based testing with dosimetrists (n = 7) using ten high-fidelity synthetic treatment plans with embedded errors; and (3) six weeks of near-live testing with dosimetrists, physicists, and trainees (n = 15–21) with weekly iterative refinement. Reported usability, usefulness, and safety issues decreased by 49% between think-aloud cohorts, and perceived usability met recommended standards (System Usability Scale > 80). Simulation-based testing was feasible, and usability and performance differed significantly between easy and hard plans. Near-live testing surfaced predominantly deeper usefulness and safety issues (77% of 142 codes) and achieved adoption rates of 49–67%. The framework provided complementary, progressively deeper insights and readied the enhanced DQC for clinical implementation.