From Prediction to Agency: A Constrained Decision Framework and Governance Stack for Agentic AI in Clinical Diagnostics

Clinical artificial intelligence systems are transitioning from predictive tools that generate diagnostic outputs for human interpretation to agentic systems capable of autonomous multistep action within clinical workflows, including ordering laboratory tests, initiating medication reconciliation, and updating patient records. Existing trust frameworks, designed for advisory systems and built on output verification and confidence calibration, do not address the governance requirements of autonomous action. We identify an agency gap: the structural mismatch between validated predictions and unvalidated action policies. Using a partially observable constrained decision process (PO-CDP) formalism, we establish the principle of agency nontransferability, demonstrating that trust calibrated at the diagnostic level does not imply safe or appropriate action policies under real-world clinical, institutional, and legal constraints. To address this gap, we propose a three-layer governance stack—epistemic soundness, policy safety, and institutional traceability—that provides verifiable guarantees at each stage of the agentic decision pipeline. This paper presents a theoretical governance framework; the phasespecific milestones in the backcasting roadmap define the empirical validation agenda for each deployment stage. A compositional risk analysis formally predicts that individually safe components can produce unsafe system-level behavior through nonlinear error propagation. An extended backcasting roadmap defines three empirically testable phases for the transition to governed agentic systems: sandboxed action proposals (2027–2029), credentialed policy systems (2030–2032), and supervised autonomy (2033–2035). The transition to agentic clinical AI constitutes a paradigm shift from prediction correctness to policy safety under constraint, requiring institutional design rather than technical improvement alone.