Data Science Statistical Significance Testing

What is a Hypothesis Test?

A statistical hypothesis is an assumption about the parameters describing a population (not a sample). This assumption may be true or false. Hypothesis tests (also called significance tests) are to accept or reject statistical hypotheses. It is often impractical to test hypotheses on the entire population, because of the size of the population, so sample data are used for the test. If a hypothesis doesn’t work for the sample, it is rejected. A hypothesis test examines two opposing hypotheses: the null hypothesis and the alternative hypothesis.

The Null Hypothesis

The null hypothesis is an assumption that there is no effect or no significant difference between specified subpopulations, any observed difference being due to chance or experimental error. The followings are some examples of the null hypothesis:

  • the average income for men is the same as that for women
  • no different profit from two different prices
  • no different number of “likes” from two sizes of posts on a SNS service

Alternative Hypothesis

The statement that is being tested against the null hypothesis is the alternative hypothesis. Researchers might hope to prove the null hypothesis wrong and their alternative hypothesis correct. For example, suppose we wanted to determine whether flipping a coin is fair. A null hypothesis might be the flips would result in Heads with 50% and Tails with 50%. The alternative hypothesis might be that the number of Heads and Tails would be very different.

Hypothesis Tests

Hypothesis testing is a formal process to determine whether to reject a null hypothesis. It consists of four steps.

  • State the null and alternative hypotheses.
  • Formulate an analysis plan about how to use sample data to evaluate the null hypothesis.
  • Analyze sample data. Find the value of the test statistic (mean score, proportion, t statistic, z-score, etc.).
  • Interpret results. Apply decision rules for rejecting the null hypothesis. In practice, the decision rules are described in two ways: 1) with reference to a P-value or 2) with reference to a region of acceptance. If the value of the test statistic is unlikely for the null hypothesis, the null hypothesis is rejected.