Ordered Piecewise Additive Learner: A Hybrid Regression Algorithm Combining Linear Basis Functions and Regression Trees

Algorithms based on gradient-boosted trees are known for predictive accuracy in tabularregression, while hybrid boosting designs adding non-tree terms remain underexplored. We in-troduce Ordered Piecewise Additive Learner (OPAL), a stagewise regression model combininghinge, categorical indicator, and regression-tree terms. Stage 1 of OPAL sequentially adds hingeand categorical indicator terms to squared loss residuals; Stage 2 fits regression-tree terms toremaining residual structure. To increase expressiveness, OPAL augments data with determinis-tic numeric transformations, including products, ratios, differences, signed square roots, signedlogarithm of one plus absolute value transforms, and squared terms. OPAL can optionally fita single tree before the two-stage model and train the main component on the resulting residu-als. As regularization, inner cross-validation selects the prediction rule among the full predictor,the two-stage component alone, the optional single-tree component, the training-set mean, or aweighted blend. The rule is stored as fitted state, and the selected rule is reflected in a distilledadditive prediction equation used for term-level summaries. Across 35 regression tasks, OPALhas the best average RMSE rank against XGBoost, LightGBM, and MARS: 1.8571 versus 2.0286,2.4286, and 3.6857. A Friedman test rejects equality of ranks (χ2F = 42.977, p = 2.49 ×10−9).Log-ratio paired tests do not separate OPAL from XGBoost but favor OPAL over LightGBMand MARS. The results support the strong predictive performance of the configured OPALpipeline as a whole rather than any isolated algorithmic component; potential research direc-tions therefore include further study of OPAL’s constituent parts. A unique aspect of OPAL isits partial additive transparency: hinge and categorical terms support effect-curve summarieswhen they have nonzero weight in the distilled prediction equation, although the full predictormay include generated pairwise features and tree terms involving the same raw variables.