Existing vision-based defect detection algorithms built upon YOLOv11 often exhibit unstable performance in complex building environments, where varying illumination conditions and partial occlusions caused by debris or vegetation can severely degrade detection accuracy. More importantly, most existing methods rely solely on visual features while neglecting domain-specific prior knowledge from civil engineering, particularly the geometric continuity of structural damages and the physical stress distribution around defect regions. As a result, these approaches remain vulnerable to background interference, show limited capability in extracting features of small-scale defects, and may generate detections that are inconsistent with the actual physical characteristics of structures.To overcome these limitations, this paper proposes an enhanced detection framework, termed **PIA-YOLO**, which integrates a Physical Information Attention (PIA) module and a Residual Efficient Channel Attention (RECA) module as dual attention branches. Specifically, the PIA module incorporates civil engineering priors by embedding physically inspired gradient operators into the attention mechanism, rather than directly solving physical equations, thereby enhancing structural feature perception and suppressing physically unreasonable detections. Meanwhile, the RECA module adaptively recalibrates channel-wise feature responses through learnable residual coefficients, enabling more effective representation of subtle defects such as cracks and spalling that are characterized by small targets and weak pixel contrast.Extensive experiments on both public datasets and a self-built crack dataset demonstrate the effectiveness of the proposed method. Compared with the baseline YOLOv11, PIA-YOLO improves mAP@0.5 by 2.2\% and 15.9\%, respectively, while increasing recall by 4.6\% and 34.0\%, without significantly sacrificing inference speed or increasing computational cost. These results indicate that PIA-YOLO provides an efficient and accurate solution for intelligent building defect detection, with promising applications in structural inspection, environmental monitoring, traffic infrastructure management, and post-disaster assessment.