Machine Learning Statistics (Tree Models)

Tree models are an important family of machine learning methods in data science used to classify observations and predict numerical outcomes by learning decision rules from data. This topic introduces students to the fundamental principles of decision trees, in which data are divided into smaller groups based on predictor variables and selected splitting criteria. Students will learn the main components of a decision tree, including the root node, internal nodes, branches, and leaf nodes, as well as how a tree makes predictions through a sequence of decision rules. Practical examples, such as classifying patients according to disease risk, predicting student performance, and identifying factors associated with environmental quality, will help students understand how tree models transform complex datasets into interpretable predictive rules.

The material focuses on classification trees and regression trees, which are designed for categorical and numerical outcomes, respectively. Students will explore how classification trees select splits using measures such as Gini impurity and entropy, while regression trees commonly use reductions in squared error or variance to determine optimal splits. The topic also introduces important concepts such as tree depth, minimum samples per leaf, pruning, and stopping criteria, which influence model complexity and predictive performance. Students will learn how shallow trees may underfit data, whereas overly complex trees may overfit the training observations. In addition, the material will provide an overview of ensemble tree methods, particularly random forests, which combine multiple decision trees to improve predictive stability and accuracy. Through these concepts, students will understand the strengths and limitations of individual trees and the benefits of combining multiple models.

Furthermore, students will learn to implement and evaluate tree-based models using R and Python. The material covers data preparation, model training, prediction, visualization of decision trees, and interpretation of variable importance. Students will assess classification performance using confusion matrices, accuracy, precision, recall, and other appropriate metrics, while regression models will be evaluated using measures such as mean absolute error (MAE) and root mean squared error (RMSE). They will also learn to divide datasets into training and testing sets, use cross-validation to tune model parameters, and compare tree models with alternative machine learning methods. By the end of this topic, students are expected to develop, interpret, and evaluate tree-based models for practical prediction problems while understanding their advantages, limitations, and potential applications in statistical analysis and data science.