Statistical Significance: What a Test Shows
Statistical significance means a test result meets a chosen threshold for rejecting a null hypothesis under its statistical model; the usual rule compares the p-value with the significance level, alpha.
The American Statistical Association’s guidance distinguishes that test result from the probability that a hypothesis is true, the size of an effect, and the importance of a finding.
What a significance test evaluates
A significance test starts with a null hypothesis, written as H0, and an alternative hypothesis, written as H1 or Ha; NIST’s explanation of statistical tests describes these as the propositions the procedure evaluates.
The null hypothesis can state that population means or proportions are equal, or that a population value matches a specified reference.
The alternative determines the direction of the test: a two-sided test looks for a departure in either direction, while a one-sided test looks in a specified direction, following NIST’s distinction between alternatives.
What a p-value means
A p-value is the probability of obtaining a test statistic at least as extreme as the observed statistic, assuming the null hypothesis and the test’s model hold; NIST defines it as a probability calculated under the null.
The direction matters when calculating that probability: NIST’s two-proportion test documentation uses different tail probabilities for lower-sided, upper-sided, and two-sided alternatives.
A p-value does not give the probability that H0 is true, that the alternative is true, or that chance alone produced the data, according to the ASA’s principles for interpreting p-values; subtracting it from one therefore does not give the probability that the finding is correct.
What the 0.05 significance level means
The significance level, alpha, is a threshold selected before interpreting the result; NIST identifies 0.10, 0.05, and 0.01 as common choices, without treating any one of them as a universal requirement.
Under the decision rule described in Penn State’s hypothesis-testing lesson, the comparison works as follows:
| Comparison with the chosen alpha | Test decision | Description |
|---|---|---|
| p-value at or below alpha | Reject H0 | Statistically significant |
| p-value above alpha | Fail to reject H0 | Not statistically significant |
Alpha of 0.05 sets a nominal 5% risk of rejecting a true null, as NIST explains; it describes the procedure under its assumptions rather than the probability that a particular significant finding is wrong.
The threshold should reflect the question and the consequences of an incorrect decision, as the ASA president’s task force recommends. The ASA’s p-value statement also warns against making conclusions depend solely on crossing a cutoff.
How statistical significance is calculated
The calculation depends on what is measured and how observations are related: NIST’s account of t-tests distinguishes paired from unpaired mean comparisons and equal from unequal variance assumptions, with different calculations for those variants.
For a one-sample mean test with an estimated population standard deviation, NIST gives the t statistic as:
t = (sample mean - hypothesized mean) / (sample standard deviation / sqrt(n))
Here, n is the sample size and the denominator is the estimated standard error of the mean; the statistic is evaluated using a t distribution with n - 1 degrees of freedom in NIST’s procedure.
For two binomial proportions, NIST documents a different large-sample test that uses the observed proportions, their sample sizes, and a pooled proportion to calculate a z statistic under a normal approximation; these different calculations show why one numerical formula cannot apply unchanged to every significance test.
Define the hypotheses, calculate the appropriate test statistic and p-value, then compare with alpha: NIST connects p-values and critical regions as equivalent ways of making the test decision.
Statistical significance versus practical significance
Practical significance asks whether an effect is large enough to matter in its context; Penn State distinguishes it from statistical significance and explains that effect size describes a difference’s magnitude, while statistical significance is affected by sample size.
Reading the effect’s size and meaning alongside the test result follows the ASA’s distinction between statistical significance and a finding’s importance.
What confidence intervals add
A confidence interval gives a lower and upper limit for an estimated parameter; NIST explains that its width indicates estimation uncertainty, with a narrower interval representing greater precision.
A frequentist 95% confidence interval does not assign a 95% probability to the true value lying inside the particular interval already calculated: NIST interprets 95% confidence through repeated sampling, in which about 95% of intervals constructed by the method cover the true parameter when its assumptions hold.
When the interval and a two-sided test use matching methods, excluding the hypothesized value corresponds to rejecting that value at the matching significance level, as NIST shows for a mean test and its confidence interval; the interval additionally shows an estimated range rather than only the threshold decision.
Sample size, errors, and statistical power
Statistical tests can make two kinds of mistakes, described in NIST’s explanation of testing errors:
- Type I error: rejecting H0 when it is true, often called a false positive.
- Type II error: failing to reject H0 when it is false, often called a false negative.
Statistical power is the probability of rejecting a false null for a specified alternative; Penn State defines it as 1 - beta, where beta is the Type II error probability.
Increasing sample size can increase power and make a very small difference statistically significant without making the underlying effect larger, as Penn State explains in its sections on power and practical significance.
There is no universal minimum sample size for statistical significance: NIST’s sample-size guidance ties the requirement to the difference to detect, variability, alpha, and the tolerated Type II error risk, so planning requires a target effect and detection probability rather than a fixed number of observations that guarantees significance.
What significant and non-significant results cannot establish
A non-significant result means insufficient evidence to reject H0 at the chosen threshold, rather than proof of equality or an absent effect.
Statistical significance also does not establish causation: NIST distinguishes an association from a causal relationship, so a significant association still needs a separate justification for a cause-and-effect interpretation.
Neither label guarantees replication: the ASA president’s task force identifies study design, insufficient data, model choice, and selective reporting as sources of replication problems, including sources of uncertainty an analysis may not address.
Multiple comparisons and repeated testing
Testing many comparisons creates more opportunities for false positives: NIST explains that repeating unadjusted pairwise tests does not preserve the overall significance level of a single comparison and describes methods such as Tukey’s and Bonferroni procedures for accounting for sets of comparisons.
Repeatedly analyzing accumulating data and stopping at significance also changes the procedure: FDA’s guidance on adaptive designs explains how unadjusted interim tests can inflate Type I error and describes sequential methods that account for interim and final testing and stopping rules.
How to read and report a significance result
Interpretation requires the question, estimated effect, uncertainty, and analysis context together; the ASA president’s task force calls for clear descriptions of methods and sources of variation, including limitations the analysis has not addressed.
When reading or preparing a report, check:
- The null and alternative hypotheses, including the test direction.
- The outcome, sample sizes, and relevant study design.
- The estimated effect in meaningful units and its confidence interval.
- The test used, exact p-value, and prespecified alpha.
- Multiple comparisons, interim analyses, and departures from the analysis plan.