Machine Learning Statistics (K-Nearest Neighbors)

K-Nearest Neighbors (KNN) is a fundamental machine learning method used in data science to classify observations or predict numerical outcomes based on the characteristics of their nearest neighbors. This topic introduces students to the principles of supervised learning and the concept that observations with similar characteristics tend to have similar outcomes. Students will learn how KNN uses existing labeled data to make predictions for new observations by identifying the K closest data points in the feature space. The material will distinguish between KNN classification, which assigns observations to categories, and KNN regression, which predicts numerical values. Practical examples, such as classifying patients according to disease risk or predicting housing prices based on similar properties, will help students understand the applications of KNN in real-world problems.

The material covers the fundamental components of the KNN algorithm, including the selection of K, distance measurement, neighbor identification, and prediction. Students will explore common distance metrics, particularly Euclidean and Manhattan distances, and understand how these measures determine the similarity between observations. The topic will also explain the importance of data preprocessing, especially feature scaling and normalization, because variables measured on different scales can disproportionately influence distance calculations. Students will investigate how the choice of K affects model complexity and prediction accuracy, including the potential for overfitting when K is too small and underfitting when K is too large. Through practical examples, students will learn how to select an appropriate value of K, including the use of cross-validation, and understand the computational characteristics of this instance-based learning method.

Furthermore, students will learn to implement and evaluate KNN models using R and Python. The material will cover data preparation, feature selection, model training, prediction, and performance evaluation using suitable metrics. For classification, students will examine confusion matrices, accuracy, sensitivity, specificity, precision, and recall, while regression performance may be assessed using measures such as mean absolute error (MAE) and root mean squared error (RMSE). Students will also learn to divide data into training and testing sets and use cross-validation to assess model performance and select an appropriate value of K. By the end of this topic, students are expected to develop, interpret, and evaluate KNN models, understand their strengths and limitations, and apply this method to practical classification and prediction problems in data science.