As key lifting equipment in construction engineering, tower cranes (TCs) play a critical role in prefabricated buildings (PBs). However, current construction scheduling relies primarily on manual observation by operators and assistants and their experience to perform repetitive tasks, resulting in inefficiency, tediousness, and safety hazards. To enhance lean construction and management efficiency in PBs, this study proposes a scheduling model that comprehensively considers the initial hook position and the specific locations of prefabricated component (PC) supply and demand points. The model is then solved using particle swarm optimization (PSO). Optimization results clearly show that the operational times of two TCs are reduced by 23.94% and 12.16%, respectively, while their daily operating costs decrease by ¥207.29 and ¥293.96. Moreover, the overall construction cost of the PBs is lowered by 8.0%. These findings clearly demonstrate the effectiveness of the proposed model in significantly improving construction efficiency and promoting lean management in PBs.