This paper addresses the robust trajectory tracking problem of an Unmanned Aerial Vehicle (UAV) equipped with a 2-DOF manipulator, designed for fast aerial manipulation of varying payloads. To overcome the high computational cost and adaptability limitations of traditional model-based controllers, this work introduces a novel hybrid gain-scheduling framework that shifts the computational complexity to the pre-flight phase. The approach utilizes an approximate inverse dynamics linearization, based on fixed nominal models, which transforms the complex nonlinear system into a simple linear plant with bounded, structured uncertainties. The entire configuration space, including manipulator states and a range of payload properties, is partitioned into dynamically similar regions using K-Means clustering. For each local region, a dedicated robust PD controller is designed using a multi-objective Genetic Algorithm (GA). This framework also successfully implements a gain interpolation technique to mitigate the potential for abrupt control actions. Simulation results validate the controller’s ability to maintain high-precision tracking during fast maneuvers and payload switching, confirming the robustness and adaptability of the offline-tuned design.