Almost sure hypothesis testing

In statistics, almost sure hypothesis testing or a.s. hypothesis testing utilizes almost sure convergence in order to determine the validity of a statistical hypothesis with probability one. This is to say that whenever the null hypothesis is true, then an a.s. hypothesis test will fail to reject the null hypothesis w.p. 1 for all sufficiently large samples. Similarly, whenever the alternative hypothesis is true, then an a.s. hypothesis test will reject the null hypothesis with probability one, for all sufficiently large samples. Along similar lines, an a.s. confidence interval eventually contains the parameter of interest with probability 1. Dembo and Peres (1994) proved the existence of almost sure hypothesis tests.

Description

For simplicity, assume we have a sequence of independent and identically distributed normal random variables, $\textstyle x_{i}\sim N(\mu ,1)$ , with mean $\textstyle \mu$ , and unit variance. Suppose that nature or simulation has chosen the true mean to be $\textstyle \mu _{0}$ , then the probability distribution function of the mean, $\textstyle \mu$ , is given by

\Pr(\mu \leq t)=[t\in [\mu _{0},+\infty ]]

where an Iverson bracket has been used. A naïve approach to estimating this distribution function would be to replace true mean on the right hand side with an estimate such as the sample mean, $\textstyle {\hat {\mu }}$ , but

{\begin{aligned}&\operatorname {E} \left[t\in \left[{\widehat {\mu }},+\infty \right]\right]=\Pr({\widehat {\mu }}\leq t)\\[4pt]={}&\Phi ({\sqrt {n}}(t-\mu _{0}))\rightarrow \Pr(\mu \leq t)-0.5[\mu _{0}=t]\end{aligned}}

which means the approximation to the true distribution function will be off by 0.5 at the true mean. However, $\textstyle \left[{\widehat {\mu }},+\infty \right]$ is nothing more than a one-sided 50% confidence interval; more generally, let $\textstyle Z_{\alpha _{n}}$ be the critical value used in a one-sided $\textstyle 1-\alpha _{n}$ confidence interval, then

\operatorname {E} \left[t\in \left[{\hat {\mu }}-{\frac {Z_{\alpha _{n}}}{\sqrt {n}}},+\infty \right]\right]\rightarrow \Pr(\mu \leq t)-\lim _{n\rightarrow +\infty }\alpha _{n}[\mu _{0}=t]

If we set $\textstyle \alpha _{n}=0.05$ , then the error of the approximation is reduced from 0.5 to 0.05, which is a factor of 10. Of course, if we let $\textstyle \alpha _{n}\rightarrow 0$ , then

\operatorname {E} \left[t\in \left[{\widehat {\mu }}-{\frac {Z_{\alpha _{n}}}{\sqrt {n}}},+\infty \right]\right]\rightarrow \Pr(\mu \leq t)

However, this only shows that the expectation is close to the limiting value. Naaman (2016) showed that setting the significance level at $\textstyle \alpha _{n}=n^{-p}$ with $\textstyle p>1$ results in a finite number of type I and type II errors w.p.1 under fairly mild regularity conditions. This means that for each $\textstyle t$ , there exists an $\textstyle N(t)$ , such that for all $\textstyle n>N(t)$ ,

\left[t\in \left[{\widehat {\mu }}-{\frac {Z_{\alpha _{n}}}{\sqrt {n}}},+\infty \right]\right]=\Pr(\mu \leq t)

where the equality holds w.p. 1. So the indicator function of a one-sided a.s. confidence interval is a good approximation to the true distribution function.

Applications

Optional stopping

For example, suppose a researcher performed an experiment with a sample size of 10 and found no statistically significant result. Then suppose she decided to add one more observation, and retest continuing this process until a significant result was found. Under this scenario, given the initial batch of 10 observations resulted in an insignificant result, the probability that the experiment will be stopped at some finite sample size, $N_{s}$ , can be bounded using Boole's inequality

\Pr(N_{s}<+\infty )<\sum _{n=11}^{+\infty }\alpha _{n}<0.0952

where $\alpha _{n}=n^{-2}$ . This compares favorably with fixed significance level testing which has a finite stopping time with probability one; however, this bound will not be meaningful for all sequences of significance level, as the above sum can be greater than one (setting $\alpha _{n}=n^{-1.2}$ would be one example). But even using that bandwidth, if the testing was done in batches of 10, then

\Pr \left(N_{s}<+\infty \right)<\sum \limits _{i=2}^{\infty }\left(10i\right)^{-1.2}<0.3

which results in a relatively large probability that the process will never end.

Publication bias

As another example of the power of this approach, if an academic journal only accepts papers with p-values less than 0.05, then roughly 1 in 20 independent studies of the same effect would find a significant result when there was none. However, if the journal required a minimum sample size of 100 and a maximum significance level is given by $\alpha _{n}<n^{-1.2}$ , then one would expect roughly 1 in 250 studies would find an effect when there was none (if the minimum sample size was 30, it would still be 1 in 60). If the maximum significance level was given by $\alpha _{n}<n^{-2}$ (which will have better small sample performance with regard to type I error when multiple comparisons are a concern), one would expect roughly 1 in 10000 studies would find an effect when there was none (if the minimum sample size was 30, it would be 1 in 900). Additionally, A.S. hypothesis testing is robust to the multiple comparisons.

Jeffreys–Lindley paradox

Lindley's paradox occurs when

The result is "significant" by a frequentist test, at, for example, the 5% level, indicating sufficient evidence to reject the null hypothesis, and
The posterior probability of the null hypothesis is high, indicating strong evidence that the null hypothesis is in better agreement with the data than is the alternative hypothesis.

However, the paradox does not apply to a.s. hypothesis tests. The Bayesian and the frequentist will eventually reach the same conclusion.

References

Naaman, Michael (2016). "Almost sure hypothesis testing and a resolution of the Jeffreys–Lindley paradox". Electronic Journal of Statistics. 10 (1): 1526–1550.
Dembo, Amir; Peres, Yuval (1994). "A topological criterion for hypothesis testing". The Annals of Statistics. 22 (1): 106–117.