Best Practices for Statistical Significance Testing
Misunderstanding statistical significance leads to wrong conclusions. Learn proper hypothesis testing and common pitfalls to avoid.
Key Takeaways
- A p-value is the probability of observing results at least as extreme as the data, assuming the null hypothesis is true.
- The conventional threshold is p < 0.05, meaning a 5% false positive rate.
- Small samples produce unreliable results.
- Testing 20 hypotheses at p < 0.05 guarantees approximately one false positive by chance alone.
- A statistically significant result may not be practically important.
Percentage Calculator
Calculate percentages, increases, decreases, and ratios
What p-Value Actually Means
A p-value is the probability of observing results at least as extreme as the data, assuming the null hypothesis is true. It is NOT the probability that the hypothesis is true or false.
Choosing Significance Levels
The conventional threshold is p < 0.05, meaning a 5% false positive rate. For high-stakes decisions (medical treatments, safety systems), use p < 0.01 or p < 0.001. For exploratory analysis, p < 0.10 may be acceptable.
Sample Size Matters
Small samples produce unreliable results. A coin that lands heads 7 out of 10 times might be fair. A coin that lands heads 700 out of 1000 times is almost certainly biased. Power analysis helps determine the minimum sample size.
Multiple Comparisons Problem
Testing 20 hypotheses at p < 0.05 guarantees approximately one false positive by chance alone. Apply Bonferroni correction (divide alpha by number of tests) or use False Discovery Rate (FDR) for multiple comparisons.
Practical vs Statistical Significance
A statistically significant result may not be practically important. A drug that reduces blood pressure by 0.1 mmHg might achieve p < 0.001 with a large sample but is clinically meaningless. Always report effect sizes alongside p-values.
Herramientas relacionadas
Guías relacionadas
How to Use Scientific Notation and Number Formatting
Scientific notation makes very large and very small numbers manageable. This guide covers notation systems, significant figures, and formatting conventions used in science, engineering, and finance.
Unit Conversion Best Practices for Developers
Incorrect unit conversions have caused spacecraft crashes and medical errors. This guide covers best practices for implementing unit conversions in software, including precision handling and common pitfalls.
Percentage Calculations: Common Formulas and Pitfalls
Percentage calculations appear simple but hide common errors, especially around percentage change, percentage points, and compound percentages. This guide clarifies the math behind everyday percentage problems.
Matrix Operations: A Visual Guide for Developers
Matrices are fundamental to graphics, machine learning, and data processing. This guide explains matrix addition, multiplication, transposition, and inversion with visual examples and practical applications.
Troubleshooting Floating-Point Precision Errors
Floating-point arithmetic produces surprising results that can cause bugs in financial calculations, comparisons, and cumulative operations. This guide explains why these errors occur and how to handle them.