P-value and Statistical Significance

Introduction & the hypotheses

• Why a significance test exists — Inference is the leap from a single sample to a population parameter. Because chance alone almost guarantees that two sample means will differ, a bare observed difference proves nothing — the whole apparatus of the p-value and significance testing exists to discipline this leap and quantify how plausibly the data could have arisen by chance.
• Null hypothesis (H₀) — When two samples are compared, the hypothesis under test is that both came from the same population — i.e. there is no difference between the populations they were drawn from. This is the null hypothesis.
• Alternative / study hypothesis (H₁) — The difference one sets out to demonstrate — what is likely to arise or would be clinically worthwhile (e.g. the benefit expected from a new treatment). It is specified at the planning stage, before the data are seen.
• The logic is a disjunction — When an observed difference exceeds the set limits, we face two choices: either an unusual low-probability event has happened, or H₀ is incorrect. The coin analogy is exact — five identical tosses of a fair coin has probability 6.3%; if unwilling to believe such an unlucky event, we reject H₀.
• A test makes H₀ unlikely — never likely — A significance test tries to show H₀ is unlikely, not its converse. A difference beyond the limits makes H₀ unlikely; a difference within the limits does not make H₀ likely — the locked aphorism is “absence of evidence is not evidence of absence”.