Current management strategies for sorghum aphids heavily rely on indiscriminate chemical application, leading to severe environmental consequences and impacting food safety. While precision spraying offers a viable remediation for pesticide overuse, its effectiveness depends on accurate pest location and classification. To address the critical challenge of segmenting small, swarming aphids in complex field environments, we propose FESW-UNet, a dual-domain attention network that integrates Fourier-enhanced attention, spatial attention, and wavelet-based downsampling into a UNet backbone. We introduce an Efficient Multi-scale Aggregation (EMA) module between the encoder and decoder to improve global context perception, allowing the model to better capture relationships between global and local features in the field. In the feature extraction stage, we embed a Similarity-Aware Activation module (SimAM) to target key infestation regions while suppressing background noise, thereby enhancing pixel-level discrimination. Furthermore, we replace conventional downsampling with Haar Wavelet Decomposition (HWD) to reduce resolution while preserving structural edge details. Finally, a Fourier-enhanced attention module (FEAM) is added to the skip-connection layers. By using complex-valued weights to regulate frequency-domain features, FEAM fuses global low-frequency structures with local high-frequency details, improving feature representation diversity. Experiments on the Aphid Cluster Segmentation dataset show that FESW-UNet outperforms other models, achieving an mIoU of 68.76% and mPA of 78.19%. The model also demonstrated strong adaptability on the AphidSeg-Sorghum dataset, reaching an mIoU of 81.22% and mPA of 87.97%. The proposed method provides an efficient and feasible technical solution for monitoring and controlling sorghum aphids via image segmentation and demonstrates broad application potential.