Defect detection plays a pivotal role in quality control for fabrics. In order to enhance the accuracy and efficiency of fabric defect detection, we have proposed the PRC-Light YOLO model for fabric defect detection and have established a detection system. Firstly, we add new convolution operators in Backbone for the YOLOv7 and integrate them with Extended-Efficient Layer Aggregation Network. This combination constructs a new feature extraction module that not only reduces the computational effort of the network model, but also extracts spatial features effectively. Secondly, we employ multi-branch dilated convolutions feature pyramid, and introduce lightweight upsampling operators to improve performance of the feature fusion network. This module achieves an expanded receptive field by generating real-time adaptive convolution kernels, allowing for the collection of crucial information from regions with a richer contextual context. To further minimize the computation during network model training, we adopt the HardSwish activation function. Finally, we apply the Wise-IOU v3 bounding box loss function as a dynamic non-monotonic focusing mechanism, which decreases adverse gradients from low-quality instances. We conduct data augmentation on real fabric dataset to raise the generalization capability of the PRC-Light YOLO model. Compared with the YOLOv7 model, numerous simulation experiments show that our proposed methods reduce the model’s parameters and computation by 18.03% and 20.53%, respectively. Simultaneously, there has been a 7.6% improvement in mAP.