The accuracy of star position simulation is the key factor affecting the accuracy of ground calibration test of spacecraft attitude measurement system. To solve the impact of nonimaging polarized stray light on the simulation accuracy of star position in the optical engine of the system, the generation mechanism of non-imaging polarized stray light has been investigated, and an optimization method for polarization balance of the optical engine is proposed. The mathematical model of the incident light aperture angle and the ideal polarization parameter of the optical engine thin film is established, and the optimization evaluation function of polarization balance of the optical engine is constructed. An optical engine polarization balance film with the corresponding full-band binomial attenuation is <0.001 and the phase delay is <1°. Experimental results show that the simulation accuracy of the weak starlight simulation system is improved by 1.64 times using polarization balanced optical engine.