As a commonly used component in robot joints, helical planetary gear system is restricted from further application in the robotics industry due to excessive maximum subsurface shear stress and vibration amplitude during their meshing motion. Tooth modification can effectively reduce the maximum subsurface shear stress and vibration amplitude of gears, making it particularly important to conduct research on the modification of helical planetary gear trains. In this study, a lumped mass method is first adopted to establish a bending-torsion-axial coupling dynamic model of the helical planetary gear train. Subsequently, multi-objective optimization modification research on the left tooth flank of the planetary gear is carried out using both traditional empirical formulas and an improved MOPSO, respectively. Then, the finite element method is employed to analyze the maximum subsurface shear stress of planetary gears under three scenarios: unmodified, traditionally modified, and modified with the improved MOPSO. Finally, the 4th-order Runge-Kutta method is used to solve the bending-torsion-axial coupling dynamic model of the helical planetary gear train system, thereby obtaining the vibration amplitude of the sun gear under the three scenarios. The research results show that the empirical formula method and the improved MOPSO reduce the maximum subsurface shear stress of the planetary gear by 12.629% and 30.107%, respectively, and decrease the vibration amplitude of the sun gear by 10.26% and 19.29%, respectively. This study provides theoretical and data support for the development of helical planetary gear modification and promotes its further application in the robotics industry.