8.4.4 P-Values
In the above discussions, we only reported an "accept" or a "reject" decision as the conclusion of a hypothesis test. However, we can provide more information using what we call $P$-values. In other words, we could indicate how close the decision was. More specifically, suppose we end up rejecting $H_0$ at at significance level $\alpha=0.05$. Then we could ask: "How about if we require significance level $\alpha=0.01$?" Can we still reject $H_0$? More specifically, we can ask the following question:
P-value is the lowest significance level $\alpha$ that results in rejecting the null hypothesis.
Intuitively, if the $P$-value is small, it means that the observed data is very unlikely to have occurred under $H_0$, so we are more confident in rejecting the null hypothesis. How do we find $P$-values? Let's look at an example.
Example
You have a coin and you would like to check whether it is fair or biased. More specifically, let $\theta$ be the probability of heads, $\theta=P(H)$. Suppose that you need to choose between the following hypotheses:
$\quad$ $H_0$ (the null hypothesis): The coin is fair, i.e., $\theta=\theta_0=\frac{1}{2}$.
$\quad$ $H_1$ (the alternative hypothesis): The coin is not fair, i.e., $\theta > \frac{1}{2}$.
- Can we reject $H_0$ at significance level $\alpha=0.05$?
- Can we reject $H_0$ at significance level $\alpha=0.01$?
- What is the $P$-value?
- Solution
-
Let $X$ be the random variable showing the number of observed heads. In our experiment, we observed $X=60$. Since $n=100$ is relatively large, assuming $H_0$ is true, the random variable
\begin{align}
W=\frac{X-n\theta_0}{\sqrt{n\theta_0(1-\theta_0)}}=\frac{X-50}{5}
\end{align}
is (approximately) a standard normal random variable, $N(0,1)$. If $H_0$ is true, we expect $X$ to be close to $50$, while if $H_1$ is true, we expect $X$ to be larger. Thus, we can suggest the following test: We choose a threshold $c$. If $W \leq c$, we accept $H_0$; otherwise, we accept $H_1$. To calculate $P(\textrm{type I error})$, we can write
\begin{align}%\label{}
P(\textrm{type I error})&=P(\textrm{Reject }H_0 \; | \; H_0)\\
&=P(W>c \; | \; H_0).
\end{align}
Since $W \sim N(0,1)$ under $H_0$, we need to choose
\begin{align}%\label{}
c=z_{\alpha},
\end{align}
to ensure significance level $\alpha$. In this example, we obtain
\begin{align}
W=\frac{X-50}{5}=\frac{60-50}{5}=2.
\end{align}
- If we require significance level $\alpha=0.05$, then \begin{align}%\label{} c=z_{0.05}=1.645 \end{align} The above value can be obtained in MATLAB using the following command: $\mathtt{norminv(1-0.05)}$. Since we have $W=2>1.645$, we reject $H_0$, and accept $H_1$.
- If we require significance level $\alpha=0.01$, then \begin{align}%\label{} c=z_{0.01}=2.33 \end{align} The above value can be obtained in MATLAB using the following command: $\mathtt{norminv(1-0.01)}$. Since we have $W=2 \leq 2.33$, we fail to reject $H_0$, so we accept $H_0$.
- $P$-value is the lowest significance level $\alpha$ that results in rejecting $H_0$. Here, since $W=2$, we will reject $H_0$ if and only if $c<2$. Note that $z_{\alpha}=c$, thus \begin{align}%\label{} \alpha=1- \Phi(c). \end{align} If $c=2$, we obtain \begin{align}%\label{} \alpha=1- \Phi(2)=0.023 \end{align} Therefore, we reject $H_0$ for $\alpha>0.023$. Thus, the $P$-value is equal to $0.023$.
-
Let $X$ be the random variable showing the number of observed heads. In our experiment, we observed $X=60$. Since $n=100$ is relatively large, assuming $H_0$ is true, the random variable
\begin{align}
W=\frac{X-n\theta_0}{\sqrt{n\theta_0(1-\theta_0)}}=\frac{X-50}{5}
\end{align}
is (approximately) a standard normal random variable, $N(0,1)$. If $H_0$ is true, we expect $X$ to be close to $50$, while if $H_1$ is true, we expect $X$ to be larger. Thus, we can suggest the following test: We choose a threshold $c$. If $W \leq c$, we accept $H_0$; otherwise, we accept $H_1$. To calculate $P(\textrm{type I error})$, we can write
\begin{align}%\label{}
P(\textrm{type I error})&=P(\textrm{Reject }H_0 \; | \; H_0)\\
&=P(W>c \; | \; H_0).
\end{align}
Since $W \sim N(0,1)$ under $H_0$, we need to choose
\begin{align}%\label{}
c=z_{\alpha},
\end{align}
to ensure significance level $\alpha$. In this example, we obtain
\begin{align}
W=\frac{X-50}{5}=\frac{60-50}{5}=2.
\end{align}
The above example suggests the following way to compute $P$-values:
Consider a hypothesis test for choosing between $H_0$ and $H_1$. Let $W$ be the test statistic, and $w_1$ be the observed value of $W$.
- Assume $H_0$ is true.
- The $P$-value is $P(\textrm{type I error})$ when the test threshold $c$ is chosen to be $c=w_1$.
