BART : Bayesian additive regression trees

 

 

https://projecteuclid.org/journals/annals-of-applied-statistics/volume-4/issue-1/BART-Bayesian-additive-regression-trees/10.1214/09-AOAS285.full

BART: Bayesian additive regression trees

 

 

 

0. Summary:

The paper introduces BART (Bayesian Additive Regression Trees), a Bayesian “sum-of-trees” model that combines multiple regression trees with regularization to form a nonparametric regression framework. The model is fit using a Bayesian backfitting MCMC algorithm, which allows full posterior inference. BART is demonstrated to have excellent predictive performance, handling various types of data including binary classification, and is also effective for variable selection.

1. Selling Point:

BART's key selling point is its ability to model complex functions using an ensemble of weak learners (regression trees) while maintaining regularization to avoid overfitting. It provides full posterior inference, including uncertainty intervals, and shows excellent performance on diverse datasets, often outperforming other methods like random forests or boosting.

2. Advantages:

• Nonparametric flexibility: Capable of modeling complex interactions between variables without pre-specifying the functional form.
• Full posterior inference: Provides uncertainty intervals and posterior distributions, aiding in robust decision-making.
• Variable selection: Automatically identifies important predictors using model-free variable selection.
• Application diversity: Performs well in various settings including classification, regression, and variable selection.
• Open-source software: Easily accessible via the BayesTree library in R, making it practical for widespread use.

3. Disadvantages:

• Computational intensity: The MCMC-based fitting process can be computationally expensive, especially for large datasets or models.
• Model interpretation: Like many ensemble methods, it can be challenging to interpret the resulting model compared to simpler, single-tree models.
• Tuning required: Although defaults work well, tuning parameters like the number of trees can be necessary for optimal performance, which adds complexity.

 

 

1. Purpose : 함수를 근사

2. How : sum of tree model로 함수를 근사할건데 regulazation prior을 줘서 개별 tree의 effect를 작게해서 함수를 만듦

3. Why : regularization prior을 주게 되면, This prevents any single tree from dominating the model.

-> 그게 왜 안좋은데? : 안좋다기 보다는 BART의 목적을 저해시킴. This would defeat the purpose of the additive representation, where each tree is supposed to be a weak learner, contributing only a small part to the overall prediction. The “rich structure” here refers to the model's ability to handle complex patterns by combining the outputs of several smaller trees, rather than relying on just a few large trees.

Function Approximation: The advantage of having an additive model is that by summing small trees, the model can approximate complex functions (relationships between the input and output) more flexibly and accurately. If one or more large trees dominate, this flexibility is lost because the model becomes similar to a single tree rather than an ensemble of small trees working together.

 

4. Algorithem : Gibbs sampler
4-1 The MCMC algorithm here often had poor mixing, meaning that it could get stuck in local optima (i.e., parts of the model space that are not the global best solution). In practice, this means that the algorithm would take a long time to explore all possible model configurations, leading to slow convergence and poor performance.
4-2 The backfitting MCMC algorithm iteratively adjusts each tree in the model. It fits each tree to the residuals of the previous trees in the ensemble. This process is similar in spirit to boosting, but done in a Bayesian framework. Since each tree only adjusts a small portion of the overall model at each step, the algorithm is more flexible and less likely to get stuck in poor configurations.

 

5. Applications

BART_cv(Cross validation) : RMSE가 가장 낮음