Model-based Boosting 2.0
Abstract
This is an extended version of the manuscript Torsten Hothorn, Peter Bühlmann, Thomas Kneib, Mattthias Schmid and Benjamin Hofner (2010), Model-based Boosting 2.0. Journal of Machine Learning Research, 11, 2109 – 2113; https://jmlr.org/papers/v11/hothorn10a.html.
We describe version 2.0 of the R add-on package mboost. The package implements boosting for optimizing general risk functions utilizing component-wise (penalized) least squares estimates or regression trees as base-learners for fitting generalized linear, additive and interaction models to potentially high-dimensional data.
This is an extended version of the manuscript
Torsten Hothorn, Peter Bühlmann, Thomas Kneib, Mattthias Schmid
and Benjamin Hofner (2010), Model-based Boosting 2.0. Journal of Machine Learning Research, 11,
2109 – 2113;
URL https://jmlr.org/papers/v11/hothorn10a.html.
keywords
component-wise functional gradient descent, splines, decision trees.1 Overview
The R add-on package mboost (Hothorn et al., 2026) implements tools for fitting and evaluating a variety of regression and classification models that have been suggested in machine learning and statistics. Optimization within the empirical risk minimization framework is performed via a component-wise functional gradient descent algorithm. The algorithm originates from the statistical view on boosting algorithms (Friedman et al., 2000; Bühlmann and Yu, 2003). The theory and its implementation in mboost allow for fitting complex prediction models, taking potentially many interactions of features into account, as well as for fitting additive and linear models. The model class the package deals with is best described by so-called structured additive regression (STAR) models, where some characteristic of the conditional distribution of a response variable given features is modeled through a regression function of the features . In order to facilitate parsimonious and interpretable models, the regression function is structured, i.e., restricted to additive functions . Each model component might take only a subset of the features into account. Special cases are linear models , additive models , where is a function of the th feature only (smooth functions or stumps, for example) or a more complex function where is implicitly defined as the sum of multiple decision trees including higher-order interactions. The latter case corresponds to boosting with trees. Combinations of these structures are also possible. The most important advantage of such a decomposition of the regression function is that each component of a fitted model can be looked at and interpreted separately for gaining a better understanding of the model at hand.
The characteristic of the distribution depends on the measurement scale of the response and the scientific question to be answered. For binary or numeric variables, some function of the expectation may be appropriate, but also quantiles or expectiles may be interesting. The definition of is determined by defining a loss function whose empirical risk is to be minimized under some algorithmic constraints (i.e., limited number of boosting iterations). The model is then fitted using
Here , are training samples with responses and potentially high-dimensional feature vectors , and are some weights. The component-wise boosting algorithm starts with some offset for and iteratively fits residuals defined by the negative gradient of the loss function evaluated at the current fit by updating only the best model component in each iteration. The details have been described by Bühlmann and Yu (2003). Early stopping via resampling approaches or AIC leads to sparse models in the sense that only a subset of important model components defines the final model. A more thorough introduction to boosting with applications in statistics based on version 1.0 of mboost is given by Bühlmann and Hothorn (2007).
As of version 2.0, the package allows for fitting models to binary, numeric, ordered and censored responses, i.e., regression of the mean, robust regression, classification (logistic and exponential loss), ordinal regression11 1 Model family is new in version 2.0 or was added after the release of mboost 1.0., quantile††footnotemark: and expectile††footnotemark: regression, censored regression (including Cox, Weibull††footnotemark: , log-logistic††footnotemark: or lognormal††footnotemark: models) as well as Poisson and negative binomial regression††footnotemark: for count data can be performed. Because the structure of the regression function can be chosen independently from the loss function , interesting new models can be fitted (Kneib et al., 2009, e.g., in geoadditive regression,).
2 Design and Implementation
The package incorporates an infrastructure for representing loss functions (so-called ‘families’), base-learners defining the structure of the regression function and thus the model components , and a generic implementation of component-wise functional gradient descent. The main progress in version 2.0 is that only one implementation of the boosting algorithm is applied to all possible models (linear, additive, tree-based) and all families. Earlier versions were based on three implementations, one for linear models, one for additive models, and one for tree-based boosting. In comparison to the 1.0 series, the reduced code basis is easier to maintain, more robust and regression tests have been set-up in a more unified way. Specifically, the new code basis results in an enhanced and more user-friendly formula interface. In addition, convenience functions for hyper-parameter selection, faster computation of predictions and improved visual model diagnostics are available.
Currently implemented base-learners include component-wise linear models (where only one variable is updated in each iteration of the algorithm), additive models with quadratic penalties (Schmid and Hothorn, 2008, e.g., for fitting smooth functions via penalized splines, varying coefficients or bi- and trivariate tensor product splines,), and trees.
As a major improvement over the 1.0 series, computations on larger data sets (both with respect to the number of observations and the number of variables) are now facilitated by memory efficient implementations of the base-learners, mostly by applying sparse matrix techniques (Bates et al., 2026, package Matrix,) and parallelization for a cross-validation-based choice of the number of boosting iterations per default via package parallel. A more elaborate description of mboost 2.0 features is available from the mboost vignette22 2 Accessible via vignette("mboost", package = "mboost")..
3 User Interface by Example
We illustrate the main components of the user-interface by a small example on human body fat composition: Garcia et al. (2005) used a linear model for predicting body fat content by means of common anthropometric measurements that were obtained for healthy German women. In addition, the women’s body composition was measured by Dual Energy X-Ray Absorptiometry (DXA). The aim is to describe the DXA measurements as a function of the anthropometric features. Here, we extend the linear model by i) an intrinsic variable selection via early stopping, ii) additional terms allowing for smooth deviations from linearity where necessary (Kneib et al., 2009, by means of penalized splines orthogonalized to the linear effect,), iii) a possible interaction between two variables with known impact on body fat composition (hip and waist circumference) and iv) utilizing a robust median regression approach instead of risk. For the data (available as data frame bodyfat), the model structure is specified via a formula involving the base-learners corresponding to the different model components (linear terms: bols(); smooth terms: bbs(); interactions: btree()). The loss function (here, the check function for the quantile) along with its negative gradient function are defined by the QuantReg(0.5) family (Fenske et al., 2011). The model structure (specified using the formula fm), the data and the family are then passed to function mboost() for model fitting:
R> library("mboost") ### attach package `mboost'R> print(fm) ### model structure
DEXfat ~ bols(age) + bols(waistcirc) + bols(hipcirc) + bols(elbowbreadth) + bols(kneebreadth) + bols(anthro3a) + bols(anthro3b) + bols(anthro3c) + bols(anthro4) + bbs(age, center = TRUE, df = 1) + bbs(waistcirc, center = TRUE, df = 1) + bbs(hipcirc, center = TRUE, df = 1) + bbs(elbowbreadth, center = TRUE, df = 1) + bbs(kneebreadth, center = TRUE, df = 1) + bbs(anthro3a, center = TRUE, df = 1) + bbs(anthro3b, center = TRUE, df = 1) + bbs(anthro3c, center = TRUE, df = 1) + bbs(anthro4, center = TRUE, df = 1) + btree(hipcirc, waistcirc, tree_controls = ctree_control(maxdepth = 2, mincriterion = 0))
R> ### fit model for conditional median of DEXfatR> model <- mboost(fm, ### model structure+ data = bodyfat, ### 71 observations+ family = QuantReg(tau = 0.5)) ### median regressionOnce the model has been fitted it is important to assess the appropriate number of boosting iterations via the out-of-sample empirical risk. By default, bootstrap samples from the training data are drawn and the out-of-bag empirical risk is computed (parallel computation if possible):
R> ### bootstrap for assessing the `optimal' number of boosting iterationsR> cvm <- cvrisk(model, grid = 1:100 * 10)R> model[mstop(cvm)] ### restrict model to optimal mstop(cvm) iterations
Model-based BoostingCall:mboost(formula = fm, data = bodyfat, family = QuantReg(tau = 0.5)) Quantile RegressionLoss function: tau * (y - f) * ((y - f) >= 0) - (1 - tau) * (y - f) * ((y - f) < 0)Number of boosting iterations: mstop = 770Step size: 0.1Offset: 29.63Number of baselearners: 19Now, the final model is ready for a visual inspection:
R> plot(cvm); plot(model) ### depict out-of bag risk & selected componentsThe resulting plots are given in Figure 1. They indicate that a model based on three components, including a smooth function of anthro3b and a bivariate function of hip and waist circumference, provides the best characterization of the median body fat composition (given the model specification offered to the boosting algorithm). A hip circumference larger than cm leads to increased body fat but only if the waist circumference is larger than cm.
The sources of the mboost package are distributed at the Comprehensive R Archive Network under GPL-2, along with binaries for all major platforms as well as documentation and regression tests.
4 Overview on 2.0 Series Features
This additional section gives some further details on new features in the mboost 2.0 series. First of all, new families were included to permit quantile and expectile regression (QuantReg(), ExpectReg()), ordinal regression (PropOdds()), censored regression (Weibull(), Loglog(), Lognormal()) and count data regression (NBinomial()). All new families, except the quantile and expectile family, have an additional scale parameter, which is included in the family and is then subsequently estimated in the boosting algorithm without further need of modifications thereof.
For increased usability of mboost, some changes in the user interface of the base-learners were necessary. Most prominently, the center argument in bols() was renamed to intercept to reflect that it is used to specify whether the base-learner should contain an intercept (intercept = TRUE) or not. The argument by (formerly denoted by z in mboost 1.0), which can be used to specify varying coefficients, is now able to handle factors with more than two levels in addition to binary and continuous covariates. Furthermore, the base-learners bss() (smoothing spline) and bns() (penalized natural splines) are deprecated and replaced by bbs() (penalized splines), which results in qualitatively the same models but is computationally much more attractive.
The speed of model estimation and prediction has been notably improved in version 2.0. Regarding large data sets, a newly added search for duplicated observations of covariates in each base-learner leads to enormous speed-ups and far less memory consumption in the base-learner fitting in each boosting iteration. The effective number of observations in a weighted penalized least-squares problem is the number of unique covariate observations which is usually considerably smaller than the number of observations in the learning sample, especially for factors or smooth effects for millions of observations. The price to pay is an increased pre-processing time, however, pre-processing is just done once at the beginning of the algorithm.
In addition, version 2.0 of mboost offers an improved infrastructure: Firstly, the subset method model[i] can now be used to enhance or restrict a given boosting model to the specified boosting iteration i. Note that in both cases the original model will be changed to reduce the memory footprint. If the boosting model is enhanced by specifying an index that is larger than the initial mstop, only the missing i - mstop steps are fitted. If the model is restricted, the spare steps are not dropped, but only hidden from the user, i.e., if we increase i again, these boosting steps are immediately available. The newly introduced function selected() allows to inspect the path of selected base-learners and is the basis for stability selection (Meinshausen and Bühlmann, 2010, in function stabsel(), cf.). Secondly, an interface for parallel computing (based on parallel) is automatically used (if available) for cross-validation-based stopping of the algorithm in cvrisk(). With some extra work, other parallelization packages can be integrated. Moreover, cvrisk() allows for more flexible definitions of resampling schemes (-fold, cross-validation, bootstrap, subsampling). Thirdly, the predict(), plot() and coef() functions were enhanced. For linear models, the intercept is now adjusted for centered covariates and non-zero offsets are added to the intercept when using coef(..., off2int = TRUE). Predictions are computed much faster now and for all three functions an argument which was included to allow to specify which base-learner(s) should be used for prediction and plotting or which coefficients should be extracted. Per default all selected base-learners are used. Users can specify which as numeric value(s) or as a (vector of) character string(s):
R> ### new data on a grid on range(anthro3b)R> nd <- with(bodyfat, data.frame(anthro3b = seq(min(anthro3b), max(anthro3b),+ length = 100)))R> ### predictions for all base-learners of `anthro3b'R> pr <- predict(model, which = "anthro3b", newdata = nd)R> pr <- rowSums(pr) ### aggregate linear and smooth effectThe combined effect can now be used for plotting (see Figure 2).
R> plot(nd$anthro3b, pr, type = "l", xlab = "anthro3b",
+ ylab = "f(anthro3b)")
R> lines(nd$anthro3b, predict(model, which = "bols(anthro3b", newdata = nd),
+ type = "l", lty = "dashed")
References
- Matrix: sparse and dense matrix classes and methods. Note: R package version 1.7-6 External Links: Link Cited by: §2.
- Boosting algorithms: regularization, prediction and model fitting. Statistical Science 22 (4), pp. 477–505. Note: with discussion External Links: Document Cited by: §1.
- Boosting with the loss: regression and classification. Journal of the American Statistical Association 98, pp. 324–338. External Links: Document Cited by: §1, §1.
- Identifying risk factors for severe childhood malnutrition by boosting additive quantile regression. Journal of the American Statistical Association 106 (494), pp. 494–510. External Links: Document Cited by: §3.
- Additive logistic regression: a statistical view of boosting (with discussion). The Annals of Statistics 28, pp. 337–407. Cited by: §1.
- Improved prediction of body fat by measuring skinfold thickness, circumferences, and bone breadths. Obesity Research 13 (3), pp. 626–634. External Links: Document Cited by: §3.
- Mboost: model-based boosting. Note: R package version 2.9-14 External Links: Link, Document Cited by: §1.
- Variable selection and model choice in geoadditive regression models. Biometrics 65 (2), pp. 626–634. External Links: Document Cited by: §1, §3.
- Stability selection. Journal of the Royal Statistical Society: Series B (Statistical Methodology) 72 (4), pp. 417–473. Note: with discussion External Links: Document Cited by: §4.
- Boosting additive models using component-wise P-splines as base-learners. Computational Statistics & Data Analysis 53 (2), pp. 298–311. External Links: Document Cited by: §2.