Joint Direction-of-Arrival and Noise Covariance Estimation via Expectation-Maximization for Noncircular Sources in Spatially Correlated Environments

Direction-of-arrival (DOA) estimation is a fundamental problem in array signal processing with applications spanning radar, sonar, and wireless communications. Traditional subspace methods like MUSIC assume white Gaussian noise and often fail to exploit the noncircular property of many communication signals. This paper presents a tractable expectation-maximization (EM) algorithm that jointly estimates DOAs and the spatially colored noise covariance matrix while exploiting signal noncircularity through an extended observation model. We derive closed-form expressions for the E-step and M-step, establish convergence properties, and provide comprehensive performance analysis. Experimental results demonstrate that the proposed method achieves superior resolution and accuracy compared to conventional MUSIC and noncircular MUSIC, particularly in scenarios with strong spatial noise correlation. Monte Carlo simulations show RMSE improvements of up to 60% over standard methods at low SNR conditions. The algorithm successfully resolves sources separated by as little as 2 degrees with 100% detection rate, significantly outperforming existing techniques.