Statistical Experiments and Significance Tests (T-Test and Multiple Testing)

Statistical experiments and significance tests are fundamental components of data science that enable students to evaluate research questions, compare groups, and determine whether observed differences or relationships are supported by statistical evidence. This topic introduces the principles of statistical hypothesis testing, including the formulation of null and alternative hypotheses, selection of significance levels, calculation of test statistics, interpretation of p-values, and understanding of Type I and Type II errors. Students will also learn the importance of study design, sampling variability, and statistical assumptions in drawing valid conclusions from data. These concepts provide a foundation for making evidence-based decisions and distinguishing meaningful statistical findings from differences that may occur by chance.

The material focuses on t-tests as essential methods for comparing means between groups or against a reference value. Students will learn about the one-sample t-test, independent-samples t-test, and paired-samples t-test, including their assumptions, appropriate applications, and interpretation of results. Practical examples will demonstrate how these tests can be used to evaluate a sample mean against a hypothesized value, compare outcomes between two independent groups, and assess changes in the same participants before and after an intervention. Students will also practice conducting t-tests using R and Python, examining test statistics, degrees of freedom, confidence intervals, and p-values to determine whether the available evidence supports rejecting the null hypothesis.

Furthermore, this topic introduces multiple testing, which occurs when several statistical hypotheses are tested within the same analysis or study. Students will explore why conducting multiple tests increases the probability of obtaining false-positive findings and learn common approaches to controlling this risk, including the Bonferroni correction and the Holm procedure. The material will distinguish between controlling the family-wise error rate and other approaches, such as the Benjamini–Hochberg procedure for controlling the false discovery rate. Through computational exercises using R and Python, students will learn to perform multiple comparisons, adjust p-values when appropriate, and interpret results carefully. By the end of the topic, students are expected to conduct and interpret basic significance tests, select appropriate t-tests, and apply suitable multiple-testing procedures to support reliable statistical conclusions.