8  Hypothesis Testing

---

A hypothesis test turns a question about the population into a decision: given the data, do we have enough evidence to reject a specific value for the parameter?

This chapter asks three questions.

• How do we decide whether to reject a value?
  • Impose the null: this is a new use of the bootstrap, which centers the resampled statistics on the value and checks how unusual the observed statistic is. It is an alternative to inverting a CI.
  • Level \(\alpha\): this sets the coverage \(1-\alpha\) of the interval, so a smaller \(\alpha\) gives a wider interval and makes rejection harder.
• How strong is the evidence against that value?
  • Two-sided and one-sided \(p\)-values: these give the probability, if the null is true, of a statistic at least as extreme as the one observed, looking in both tails or in one.
  • \(t\)-value: this measures the distance from the null in standard errors, \(t=(M-m_{0})/\text{SE}(M)\), so it does not depend on units.
• What does “statistically significant” mean?
  • False rejections: when the null is true, \(p\)-values still vary from sample to sample, and some fall below \(0.05\) by chance. So “significant” says only that the \(p\)-value fell below a chosen cutoff.

8.1 Testing Methods

In this chapter, we test hypotheses using data-driven methods that assume much less about the data generating process. There are two main ways to conduct a hypothesis test: inverting a confidence interval, as in the previous chapter, and imposing the null. The first treats the distribution of estimates directly; the second explicitly enforces the null hypothesis to evaluate how unusual the observed statistic is. Both approaches rely on the bootstrap to approximate sampling variability.

Before either test can run, we have to name the claim being tested and the alternative we would favor if the data spoke against it (Neyman and Pearson 1933).

ImportantKey Definition

The null hypothesis \(H_{0}\) is the specific claim about the parameter that we will test.

The alternative hypothesis \(H_{A}\) is the competing claim.

A two-sided hypothesis test covers deviations from the null in either direction (alternative values are higher or lower than the null).

This setup is useful because the test only ever asks about \(H_{0}\). Either the data contradict \(H_{0}\) enough to reject it, or they do not. The data never “prove” \(H_{A}\). The most common test concerns the mean, \(H_{0}: \mathbb{E}[X_{i}] = m_{0}\), where \(m_{0}\) is a hypothesized value. The two-sided alternative is \(H_{A}: \mathbb{E}[X_{i}] \neq m_{0}\). E.g., you hypothesize your stock returns have earnings \(m_{0}=0\) and consider that the returns could truly be either positive or negative.

Choosing a Null

Where does a hypothesized value like \(m_{0}\) come from? It comes from outside the sample, and it is best written down before looking at the data. Common sources include

• a claim someone else makes that you want to check, such as a firm’s advertised battery life
• an economic theory, such as the efficient markets hypothesis, which predicts that the average return of a trading strategy, minus the market’s return, is zero
• a policy target or legal standard, such as a central bank’s \(2\%\) inflation target
• an earlier value, such as last year’s average wage
• “no effect” or “no change”, such as a mean change in store sales of zero after a new layout, which is the most common null in practice

For example, Adamson and Kimbrough (2023) and Adamson and Wilson (2024) test whether the estimates from their laboratory experiments equal the values an economic model predicts, rather than zero (Lehmann and Romano 2005).

A null chosen after seeing the data is circular: you could always pick a value that the data already contradict and then reject it.

Whatever its source, the null makes a specific claim, such as a single value for the mean. Because the claim is specific, we can compute how the statistic would vary if the null were true. The alternative covers a whole range of values, so there is no single sampling distribution to compute under it. This is also why the claim you hope to support usually goes in \(H_{A}\): the only strong conclusion a test can deliver is to reject \(H_{0}\).

Impose the Null

We can also compute a null distribution: the sampling distribution of the statistic under the null hypothesis (assuming your null hypothesis was true). We use the bootstrap to loop through a large number of “resamples”. In each iteration of the loop, we impose the null hypothesis and re-estimate the statistic of interest. We then calculate the range of the statistic across all resamples and compare how extreme the original value we observed is.

ImportantKey Definition

The rejection region is the set of values for the test statistic (or hypothesized parameter) that lead us to reject \(H_{0}\). For a two-sided test at level \(\alpha=0.05\), the rejection region is the two tails outside the central \(95\%\) of the null distribution.

The rejection region is useful as a decision rule that mirrors how we invert a CI:

• reject \(H_{0}\) if the observed statistic falls in the rejection region
• fail to reject \(H_{0}\) if it falls inside the central \(95\%\)

A \(95\%\) confidence interval corresponds to a \(5\%\) rejection region split between the two tails.

For example, suppose you hypothesize the mean equals \(m_{0}\). You then construct a \(95\%\) confidence interval around the null bootstrap distribution (resamples centered around \(m_{0}\)). Suppose your sample mean falls outside of that interval. Then even after accounting for sampling variability (which you estimate), it seems extremely unlikely that the theoretical mean actually equals \(m_{0}\), so you reject that hypothesis. (If the sample mean landed in the interval, you would “fail to reject” the theoretical mean equals \(m_{0}\).)

## MAIN EXAMPLE
# Pretend these 3294 hourly wages are the whole population
# Source: US workers, with hourly wages in 1980 dollars
data(Wages1, package='Ecdat')
population_wages <- Wages1[, 'wage']

# Lab data: one random sample of 50 workers, as if it were all we observed
set.seed(1) # to be replicable
x_hat <- sample(population_wages, 50, replace=FALSE)
m_hat <- mean(x_hat)
m_hat
## [1] 5.708014
# The sample mean is about 5.71 dollars an hour.

# Bootstrap NULL: mean=6.5
# Bootstrap shift: center each bootstrap resample so that the distribution satisfies the null hypothesis on average.
null_mean <- 6.5
boot_means_null <- rep(NA, 999)
for(b in seq_along(boot_means_null)){
    x_boot <- sample(x_hat, replace=TRUE)
    m_b_hat <- mean(x_boot) + (null_mean - m_hat) # impose the null via Bootstrap shift
    boot_means_null[b] <- m_b_hat
}
hist(boot_means_null, breaks=25, border=NA, freq=FALSE,
    main=NA,
    xlab='Null Bootstrap Samples')
ci_95 <- quantile(boot_means_null, probs=c(.025, .975)) # critical region
abline(v=ci_95, col=rgb(0, 0, 0, .8), lwd=2)
abline(v=m_hat, col=rgb(1, 0, 0, .8), lwd=2)

# The middle 95% of the null distribution runs from about 5.73 to 7.26.
# The sample mean of about 5.71 falls just below it, so we reject this null.

TipTest Yourself

Repeat this test with the murder arrest rates in USArrests, again for the hypothesized mean of \(9\).

x2_hat <- USArrests[, 'Murder'] # murder arrests per 100,000
m2_hat <- mean(x2_hat)

# Bootstrap NULL: mean=9
# Bootstrap shift: center each bootstrap resample so that the distribution satisfies the null hypothesis on average.
null_mean2 <- 9
boot_means_null2 <- rep(NA, 999)
for(b in seq_along(boot_means_null2)){
    x_boot <- sample(x2_hat, replace=TRUE)
    m_b_hat <- mean(x_boot) + (null_mean2 - m2_hat) # impose the null via Bootstrap shift
    boot_means_null2[b] <- m_b_hat
}
hist(boot_means_null2, breaks=25, border=NA, freq=FALSE,
    main=NA,
    xlab='Null Bootstrap Samples')
ci2_95 <- quantile(boot_means_null2, probs=c(.025, .975)) # critical region
abline(v=ci2_95, col=rgb(0, 0, 0, .8), lwd=2)
abline(v=m2_hat, col=rgb(1, 0, 0, .8), lwd=2)

# The sample mean falls just below the middle 95%, so we reject a mean of 9.

NoteMust Know

Why does adding \((m_{0} - \hat{m})\) impose the null? A bootstrap resample of the data has a mean that varies around the sample mean \(\hat{m}\), not around the hypothesized \(m_{0}\). Adding the constant \((m_{0} - \hat{m})\) slides the whole resampling distribution over so it is centered at \(m_{0}\) instead. A shift moves the center but leaves the spread and shape unchanged, so the result is a sampling distribution with the null mean and the same variability as the data.

For example, take the three observations \(\{2, 4, 9\}\), which have mean \(\hat{m}=5\), and the null \(m_{0}=6\). Adding \(m_{0}-\hat{m}=1\) to every observation gives \(\{3, 5, 10\}\), which has mean \(6\) and the same spread.

x0_hat <- c(2, 4, 9)
m0_hat <- mean(x0_hat)
x0_null <- x0_hat + (6 - m0_hat) # shift so the mean equals 6
x0_null
## [1]  3  5 10
c(mean(x0_null), sd(x0_hat), sd(x0_null))
## [1] 6.000000 3.605551 3.605551

The Normal approximation from the previous chapter can also impose the null: center the interval at the hypothesized value \(m_{0}\) and use \(m_{0} \pm q(\alpha/2) \cdot \text{SE}(M)\), where \(\text{SE}(M)\) is estimated via bootstrap or classical formulas. (While we could also use a Null Jackknife distribution, that is rarely done in practice.) Altogether, there are two different types of confidence intervals that “impose the null”. Until you know more, a conservative rule-of-thumb is to take the larger estimate.

Types of Confidence Interval Estimates that “impose the null”

Interval	Mechanism
Bootstrap (Percentile)	randomly resample \(n\) observations with replacement and shift
Normal	assume sampling distribution is Normal and use formula \(\pm q(\alpha/2) \cdot \text{SE}\)

Coverage and Rejection

The level \(\alpha\) ties together three things: the coverage of the interval, its width, and whether we reject. The interval holds the middle \(1-\alpha\) of the null distribution, so a smaller \(\alpha\) means higher coverage. Higher coverage needs a wider interval, as in Interval Size. A wider interval leaves a smaller rejection region, so the observed statistic must be more extreme before we reject \(H_{0}\).

Two Levels and What They Imply

Level \(\alpha\)	Coverage \(1-\alpha\)	Quantiles of the null distribution	Interval width	Rejecting \(H_{0}\)
\(0.10\)	\(90\%\)	\(5^{\text{th}}\) and \(95^{\text{th}}\)	narrower	easier
\(0.01\)	\(99\%\)	\(0.5^{\text{th}}\) and \(99.5^{\text{th}}\)	wider	harder

The figure below draws both intervals on the null bootstrap distribution from above.

## MAIN EXAMPLE
# Two levels, using boot_means_null and m_hat from above
# Middle 90% (alpha=0.10) and middle 99% (alpha=0.01) of the null distribution
ci_90 <- quantile(boot_means_null, probs=c(0.05, 0.95))
ci_99 <- quantile(boot_means_null, probs=c(0.005, 0.995))

# Width, and whether the sample mean falls outside each interval
level_table <- data.frame(alpha=c(0.10, 0.01), coverage=c(0.90, 0.99),
    lower=c(ci_90[[1]], ci_99[[1]]), upper=c(ci_90[[2]], ci_99[[2]]))
level_table[, 'width'] <- level_table[, 'upper'] - level_table[, 'lower']
level_table[, 'reject'] <- m_hat < level_table[, 'lower'] | m_hat > level_table[, 'upper']
print(level_table, digits=3)
##   alpha coverage lower upper width reject
## 1  0.10     0.90  5.83  7.15  1.32   TRUE
## 2  0.01     0.99  5.57  7.57  2.00  FALSE

# Both intervals on the same null bootstrap distribution
hist(boot_means_null, breaks=25, border=NA, freq=FALSE,
    main=NA, xlab='Null Bootstrap Samples')
title('90% and 99% Intervals', font.main=1)
abline(v=ci_90, col=rgb(0, 0, 0, .8), lwd=2)
abline(v=ci_99, col=rgb(0, 0, 0, .8), lwd=2, lty=2)
abline(v=m_hat, col=rgb(1, 0, 0, .8), lwd=2)
legend('topright', legend=c('90%', '99%'),
    lty=c(1, 2), lwd=2, col=rgb(0, 0, 0, .8), bty='n')

# The 99% interval is about 2.00 wide, compared with about 1.32 for the 90% interval.
# The sample mean of about 5.71 falls outside the 90% interval, so we reject at the 10% level.
# It falls inside the 99% interval, so we fail to reject at the 1% level.
# The population mean is about 5.76, so at the 1% level we fail to reject a null that is false.

Lowering \(\alpha\) makes it rarer to reject a null that is true, but it also makes it harder to reject a null that is false. So the choice of \(\alpha\) is a trade-off, and, like the null, it is best made before looking at the data.

8.2 \(p\)-values

Reading a hypothesis test off a confidence interval is binary (reject or not); often we also want a number that describes how extreme the observed statistic is relative to the null.

ImportantKey Definition

The \(p\)-value is the probability, under the null hypothesis, of seeing a statistic at least as extreme as the one observed. Small \(p\)-values give evidence against the null; the threshold \(p \leq 0.05\) is a convention, not a law of nature. The \(p\)-value is not the probability that the null is true.

The \(p\)-value is useful as a continuous summary of how strongly the data oppose the null, whereas a confidence interval gives only a yes/no answer at a fixed coverage level. For the running mean example, we want the probability that the random variable \(M\) is at least as extreme (far from the null mean \(m_{0}\)) as our observed sample mean \(\hat{m}\).

Recall that we used the bootstrap to estimate the distribution of the sample statistic like the mean, and the null-bootstrap shifted the bootstrap to be centered at a hypothesized value. The bootstrap idea here is to approximate \(M-\mathbb{E}[X_{i}]\), the difference between the sample mean \(M\) and the unknown theoretical mean \(\mathbb{E}[X_{i}]\), with \(M^{\text{boot}}-\hat{m}\). To test \(H_{0}: \mathbb{E}[X_{i}]=m_{0}\), we recenter the bootstrap mean as \(M^{\text{boot}}_{0}=M^{\text{boot}}+(m_{0}-\hat{m})\), so \(M^{\text{boot}}_{0}-m_{0}=M^{\text{boot}}-\hat{m}\). The two-sided \(p\)-value is then \[\begin{aligned} & \text{Prob}( |M - m_{0}| \geq |\hat{m} - m_{0}| \mid \mathbb{E}[X_{i}] = m_{0} ) \\ & \approx \text{Prob}( |M^{\text{boot}}_{0}- m_{0}| \geq |\hat{m}- m_{0}| ), \end{aligned}\] which is the share of null bootstrap means at least as far from \(m_{0}\) as \(\hat{m}\). Without ties at the observed distance, this share equals \(1-\hat{F}^{|\text{boot}|}_{0}(|\hat{m}-m_{0}|)\), where \(\hat{F}^{|\text{boot}|}_{0}\) is the ECDF of \(|M^{\text{boot}}_{0}- m_{0}|\).

Using the null bootstrap distribution boot_means_null from above, we compute the two-sided \(p\)-value for the lab data.

## MAIN EXAMPLE
# Two-Sided Test, ALTERNATIVE: mean < 6.5 or mean > 6.5
# Visualize Two Sided Prob. & reject region boundary
par(mfrow=c(1, 2))
hist(boot_means_null-null_mean,
    freq=FALSE, breaks=25,
    border=NA,
    main=NA,
    xlab=expression('Null Bootstrap for M - '~m[0]))
abline(v=m_hat-null_mean, col=rgb(1, 0, 0, .8), lwd=2)
ci_95 <- quantile(boot_means_null-null_mean, probs=c(0.025, .975))
abline(v=ci_95, col=rgb(0, 0, 0, .8), lwd=2)

# Equivalent Visualization
boot_absval <- abs(boot_means_null-null_mean)
F_hat <- ecdf(boot_absval)
plot(F_hat,
    main=NA,
    xlab=expression('Null Bootstrap for |M - '~m[0]~'|'))
abline(v=abs(m_hat-null_mean), col=rgb(1, 0, 0, .8), lwd=2)
# with Two Sided Probability
p2 <- mean(boot_absval >= abs(m_hat-null_mean))
title( paste0('p=', round(p2, 3)), font.main=1)

# The two-sided p-value is about 0.04, below the 5% level.

TipTest Yourself

Repeat the two-sided \(p\)-value with the murder arrest rates, using boot_means_null2 from the Test Yourself in Impose the Null.

# Two-Sided Test, ALTERNATIVE: mean < 9 or mean > 9
# Visualize Two Sided Prob. & reject region boundary
par(mfrow=c(1, 2))
hist(boot_means_null2-null_mean2,
    freq=FALSE, breaks=25,
    border=NA,
    main=NA,
    xlab=expression('Null Bootstrap for M - '~m[0]))
abline(v=m2_hat-null_mean2, col=rgb(1, 0, 0, .8), lwd=2)
ci2_95 <- quantile(boot_means_null2-null_mean2, probs=c(0.025, .975))
abline(v=ci2_95, col=rgb(0, 0, 0, .8), lwd=2)

# Equivalent Visualization
boot_absval2 <- abs(boot_means_null2-null_mean2)
F2_hat <- ecdf(boot_absval2)
plot(F2_hat,
    main=NA,
    xlab=expression('Null Bootstrap for |M - '~m[0]~'|'))
abline(v=abs(m2_hat-null_mean2), col=rgb(1, 0, 0, .8), lwd=2)
# with Two Sided Probability
p2_2 <- mean(boot_absval2 >= abs(m2_hat-null_mean2))
title( paste0('p=', round(p2_2, 3)), font.main=1)

# The two-sided p-value is about 0.04, below the 5% level.

NoteMust Know

Continuing the exam example from the previous chapter: the student scored \(\hat{p}=34/44 \approx 77\%\) and wants to test \(H_{0}: p=0.85\). Imposing the null shifts the bootstrap distribution of the proportion correct so that it is centered at \(0.85\) instead of at \(\hat{p}\), exactly as was done for the mean.

n_questions <- 44
n_correct <- 34
outcomes <- c(rep(1, n_correct), rep(0, n_questions - n_correct))
p_hat <- mean(outcomes)

# Bootstrap NULL: p=0.85
# Bootstrap shift: center each bootstrap resample so that the distribution satisfies the null hypothesis on average.
p0 <- 0.85
boot_props_null <- rep(NA, 999)
for(b in seq_along(boot_props_null)){
    x_boot <- sample(outcomes, replace=TRUE)
    boot_props_null[b] <- mean(x_boot) + (p0 - p_hat) # impose the null via Bootstrap shift
}

# Two-Sided p-value
boot_absval <- abs(boot_props_null - p0)
p2_exam <- mean(boot_absval >= abs(p_hat - p0))

hist(boot_props_null, breaks=25, border=NA, freq=FALSE,
    main=NA, xlab='Null Bootstrap Proportion (H0: p=0.85)')
ci_95 <- quantile(boot_props_null, probs=c(.025, .975))
abline(v=ci_95, col=rgb(0, 0, 0, .8), lwd=2)
abline(v=p_hat, col=rgb(1, 0, 0, .8), lwd=2)
title( paste0('p=', round(p2_exam, 3)), font.main=1)

# The two-sided p-value is about 0.20, far above the 5% level.
# So we fail to reject p=0.85, which matches the interval in the previous chapter.

Compared with an interval, the \(p\)-value adds a precise number for how strong or weak the evidence against the null is, rather than just a reject/fail-to-reject label.

The other story, \(H_{0}: p=0.70\), can be tested the same way.

# Impose p=0.70 instead, by shifting the same resamples
p0_low <- 0.70
boot_props_null_low <- boot_props_null + (p0_low - p0)

# Two-Sided p-value
mean(abs(boot_props_null_low-p0_low) >= abs(p_hat-p0_low))
## [1] 0.2032032
# The p-value is again about 0.20, so we also fail to reject p=0.70.
# The two nulls are nearly equally far from p_hat, so their p-values match.
# The p-value is large in both directions, so neither story is ruled out.

A single \(44\)-question exam simply cannot pin down the student’s true ability closely enough to rule out either story.

You can conduct a hypothesis test using \(p\)-values instead of confidence intervals. It is common to use this decision rule:

• reject the null at the \(5\%\) level if \(p \leq 0.05\)
• fail to reject the null at the \(5\%\) level if \(p > 0.05\)

At a general level \(\alpha\), the rule \(p \leq \alpha\) is standard in statistical theory, while \(p < \alpha\) is common in applied writing. The two rules differ only when \(p\) equals \(\alpha\) exactly, so there is no real difference in practice.

TipTest Yourself

A bootstrap \(p\)-value is the share of the \(B\) draws at least as extreme as the observed statistic, so it equals \(0\) when no draw is that extreme. Some authors instead count the observed statistic as one more draw: they add \(1\) to the number of extreme draws and \(1\) to \(B\), so the \(p\)-value can never equal \(0\) (Davison and Hinkley 1997). Compute both versions for the two-sided test of a mean of \(6.5\), and again for a mean of \(8\).

# Number of null bootstrap means at least as far from the null as the sample mean
n_extreme <- sum(abs(boot_means_null-null_mean) >= abs(m_hat-null_mean))
n_draws <- length(boot_means_null)
c(usual=n_extreme/n_draws, add_one=(n_extreme + 1)/(n_draws + 1))
##      usual    add_one 
## 0.03903904 0.04000000
# The p-value moves only from 39/999 to 40/1000.

# Impose a mean of 8 instead, by shifting the same resamples
null_mean_far <- 8
boot_means_null_far <- boot_means_null + (null_mean_far - null_mean)
n_extreme_far <- sum(abs(boot_means_null_far-null_mean_far) >= abs(m_hat-null_mean_far))
c(usual=n_extreme_far/n_draws, add_one=(n_extreme_far + 1)/(n_draws + 1))
##   usual add_one 
##   0.000   0.001
# No draw is as extreme, so the usual p-value is 0.
# The add-one p-value is 1/1000, the smallest it can be with 999 draws.

With \(999\) draws, both versions reach the same decision at the \(5\%\) level. With \(49\) extreme draws, the usual version gives \(49/999 < 1/20\) and the add-one version gives \(50/1000 = 1/20\), so both reject. With \(50\) extreme draws, they give \(50/999 > 1/20\) and \(51/1000 > 1/20\), so neither rejects. This uses the rule \(p \leq 0.05\), since the add-one version can equal \(1/20\) exactly.

Caveats

Beware of a common misreading of the \(p\)-value as “the probability the null is true”. That is false. A \(p\)-value is the frequency you see something at least as extreme as your statistic under the null hypothesis.

Failing to reject \(H_{0}\) does not show that \(H_{0}\) is true (Altman and Bland 1995; Abadie 2020). The exam example above is a case in point, because the student’s true score cannot be both \(70\%\) and \(85\%\). A large \(p\)-value can simply mean that the sample is too small to tell nearby values apart. This is one reason why we say “fail to reject” rather than “accept”. Another reason is that we can “fail to reject” multiple different hypotheses, and “accepting” them all would be nonsense. E.g., we cannot accept the mean is both \(4\) and \(2\) even if both fall inside of a confidence interval. Adamson (2025) makes this point about earlier work that “ruled out” trade because a proxy for trade was statistically insignificant. As the paper puts it, “the absence of evidence is not evidence of absence.”

Often, one may also see or hear “\(p<.05\): statistically significant” and “\(p>.05\): not statistically significant”. That is decision making on purely statistical grounds, and it may or may not be suitable for your context. You simply need to know that whoever says those things is using \(5\%\) as a critical value to reject a null hypothesis.

A rejection also does not mean the null is far from the truth (McCloskey and Ziliak 1996; Wasserstein and Lazar 2016; Imbens 2021).

Running many hypothesis tests inflates the chance of a false rejection. At the \(5\%\) level, you expect roughly \(1\) in \(20\) true nulls to be rejected by chance alone, so if you test \(20\) independent nulls and find \(1\) “significant” result, that result is what you would expect even when every null is true. The fix is to correct the level (e.g., Bonferroni (1936): use \(0.05/k\) when running \(k\) tests) or to pre-specify a small set of hypotheses before looking at the data.

# Purely-Statistical Decision Making
# via Two Sided Test
if(p2 >.05){
    print('fail to reject the null that mean=6.5, at the 5% level')
} else {
    print('reject the null that mean=6.5 in favor of either <6.5 or >6.5, at the 5% level')
}
## [1] "reject the null that mean=6.5 in favor of either <6.5 or >6.5, at the 5% level"

Also note that the \(p\)-value is itself a function of data, and hence a random variable that changes from sample to sample. To see this, return to the pretend population from Sampling. We know its mean, so we can draw many samples and test the null that is actually true.

## MAIN EXAMPLE
# Known population mean, from the pretend population defined above
population_wage_mean <- mean(population_wages)

# 300 samples, each tested against the true mean
p_values <- rep(NA, 300)
for(s in seq_along(p_values)){
    x_sim <- sample(population_wages, 50, replace=FALSE)
    m_sim <- mean(x_sim)
    boot_means_null_sim <- rep(NA, 999)
    for(b in seq_along(boot_means_null_sim)){
        x_boot <- sample(x_sim, replace=TRUE)
        boot_means_null_sim[b] <- mean(x_boot) + (population_wage_mean - m_sim) # impose the (true) null
    }
    p_values[s] <- mean(abs(boot_means_null_sim-population_wage_mean) >= abs(m_sim-population_wage_mean))
}

hist(p_values, breaks=seq(0, 1, by=.05),
    freq=FALSE, border=NA, main=NA,
    xlab='p-values when the null is true')
abline(v=.05, col=rgb(0, 0, 0, .8), lwd=2)

# Share of samples that falsely reject at the 5% level
mean(p_values <= .05)
## [1] 0.08666667
# About 9% of these samples still reject at the 5% level.
# These are the false rejections described above.
# Wages are right-skewed, so with n=50 this bootstrap test is not exact.
# Its long-run share of false rejections is about 7%, not 5%.
# With only 300 samples, the share here lands a little above that.

The null is true in every one of these samples, yet the \(p\)-values spread across the whole range from \(0\) to \(1\). Given that the \(5\%\) level is somewhat arbitrary, and that the \(p\)-value both varies from sample to sample and is often misunderstood, it makes sense to give specific \(p\)-values a limited role in decision making.

NoteMust Know

The \(p\)-value also varies a little each time you rerun the bootstrap on the same data, because each run draws different resamples. This variation comes from the computer, not from the sample, and it shrinks as you use more resamples. Here we repeat the two-sided test \(300\) times on the one sample we have.

p_values_boot <- rep(NA, 300)
for(b2 in seq(p_values_boot)){
    boot_means_null_p <- rep(NA, 999)
    for(b in seq_along(boot_means_null_p)){
        x_boot <- sample(x_hat, replace=TRUE)
        m_b_hat <- mean(x_boot) + (null_mean - m_hat) # impose the null
        boot_means_null_p[b] <- m_b_hat
    }
    p_values_boot[b2] <- mean(abs(boot_means_null_p-null_mean) >= abs(m_hat-null_mean))
}

hist(p_values_boot, freq=FALSE, breaks=50,
    border=NA, main=NA)
abline(v=.05, col=rgb(0, 0, 0, .8), lwd=2)

# The p-values run from about 0.02 to 0.06, so they land on both sides of 0.05.
# Rerunning the bootstrap alone could flip this decision.

When the \(p\)-value is close to the cutoff, rerunning the bootstrap alone could flip the decision. This is one more reason not to lean on the exact \(5\%\) cutoff.

8.3 Other Statistics

\(t\)-values

Different studies have different \(\text{SE}(M)\), so the raw difference \(\hat{m} - m_{0}\) is hard to compare across studies; we need to express the deviation in standard-error units.

ImportantKey Definition

The \(t\)-value (or \(t\)-statistic) standardizes the deviation of \(\hat{m}\) from the hypothesized \(m_{0}\) by the standard error, \[t = \frac{M - m_{0}}{\text{SE}(M)}.\] Using a standardized statistic makes results comparable across studies and connects to a well-studied reference distribution (Normal in large samples, Student’s \(t\) in small).

The \(t\)-value is useful as a test statistic that does not depend on units. The units cancel between the numerator and the denominator. So a given \(t\)-value means roughly the same thing whether the data are measured in dollars or hours. For any specific sample we must approximate the standard error. Using the theory-driven approach, we compute \(\hat{t}=(\hat{m}-m_{0})/(\hat{s}_{\text{adj}}/\sqrt{n})\), with the adjusted sample standard deviation \(\hat{s}_{\text{adj}}\). Using the data-driven approach, we compute \(\hat{t}=(\hat{m}-m_{0})/(\hat{\text{SE}}^{\text{jack}})\) or \(\hat{t}=(\hat{m}-m_{0})/(\hat{\text{SE}}^{\text{boot}})\). In any case, we can use bootstrapping to estimate the variability of the \(t\) statistic, just like we did with the mean.

NoteMust Know

Take the four observations \(\{3, 5, 7, 9\}\) and the null \(m_{0}=4\). The sample mean is \(\hat{m}=6\) and the adjusted sample standard deviation is \(\hat{s}_{\text{adj}}\approx 2.58\). The theory-driven standard error is \(\hat{s}_{\text{adj}}/\sqrt{4} \approx 1.29\), so \(\hat{t} \approx (6-4)/1.29 \approx 1.55\). The sample mean is about one and a half standard errors above the null.

x0_hat <- c(3, 5, 7, 9)
se0_hat <- sd(x0_hat)/sqrt(length(x0_hat))
t0_hat <- (mean(x0_hat) - 4)/se0_hat
c(se0_hat, t0_hat)
## [1] 1.290994 1.549193

For the lab data, we compute \(\hat{t}\) with jackknife standard errors and then bootstrap its null distribution.

## MAIN EXAMPLE
#null hypothesis
null_mean <- 6.5

# t statistic
jack_means <- rep(NA, length(x_hat))
for(i in seq_along(jack_means)){
    x_jack <- x_hat[-i]
    jack_means[i] <- mean(x_jack)
}
jackknife_se <- sd(jack_means)*sqrt(length(x_hat))
t_hat <- (m_hat - null_mean)/jackknife_se

# Bootstrap Null Distribution
boot_t_null <- rep(NA, 999)
for(b in seq_along(boot_t_null)){
    x_boot <- sample(x_hat, replace=TRUE)
    m_b_hat <- mean(x_boot) + (null_mean - m_hat) # impose the null by recentering
    # Compute t stat using jackknife ses (same as above)
    jack_means_b <- rep(NA, length(x_boot))
    for(i in seq_along(jack_means_b)){
        jack_means_b[i] <- mean(x_boot[-i])
    }
    jackknife_se_b <- sd(jack_means_b)*sqrt(length(x_boot))
    t_b_hat <- (m_b_hat - null_mean)/jackknife_se_b
    boot_t_null[b] <- t_b_hat
}

# Plot the null distribution and CI
par(mfrow=c(1, 2))
hist(boot_t_null, border=NA, breaks=50,
    freq=FALSE, main=NA, xlab='Null Bootstrap for t')
abline(v=t_hat, col=rgb(1, 0, 0, .8), lwd=2)
ci_95 <- quantile(boot_t_null, probs=c(0.025, 0.975) )
abline(v=ci_95, col=rgb(0, 0, 0, .8), lwd=2)

# Compute the p-value for two-sided test
F_hat <- ecdf(abs(boot_t_null))
plot(F_hat,
    xlim=range(boot_t_null, t_hat),
    xlab='Null Bootstrap for |t|',
    main=NA)
abline(v=abs(t_hat), col=rgb(1, 0, 0, .8), lwd=2)
p <- mean(abs(boot_t_null) >= abs(t_hat))
title( paste0('p=', round(p, 3)), font.main=1)

if(p >.05){
    print('fail to reject the null that mean=6.5, at the 5% level')
} else {
    print('reject the null that mean=6.5 in favor of either <6.5 or >6.5, at the 5% level')
}
## [1] "reject the null that mean=6.5 in favor of either <6.5 or >6.5, at the 5% level"
# The observed t is about -1.99, just outside the middle 95% of its null distribution.
# Its two-sided p-value, about 0.04, is below 0.05, so this test also rejects.

There are several benefits to this statistic:

• uses the same statistic for different hypothesis tests
• makes the statistic comparable across different studies
• removes dependence on unknown parameters by normalizing with a standard error
• makes the null distribution theoretically known asymptotically (approximately)

The last point implies we are typically dealing with a normal distribution that is well-studied, or another well-studied distribution derived from it. Because \(\hat{t}\) is comparable across tests, many null distributions can share one figure. For example, Adamson (2022) summarizes many simulated null distributions of \(t\)-values (Hope 1968; Besag and Diggle 1977; Baddeley et al. 2015) with “spreadplots”. Each spreadplot draws the range as a black line, the middle \(95\%\) between the \(0.025\) and \(0.975\) quantiles as a thick grey bar, and the median as a white line. The observed \(t\)-values appear as points, with reference lines at \(0\) and \(\pm 1.96\).

TipTest Yourself

Make a spreadplot for three tests at once. In USArrests, test the nulls \(\mathbb{E}[X_{i}]=9\) for Murder, \(\mathbb{E}[X_{i}]=150\) for Assault, and \(\mathbb{E}[X_{i}]=70\) for UrbanPop. For each variable, compute \(\hat{t}\) and bootstrap its null distribution as above. Then draw one spreadplot per variable and add each observed \(\hat{t}\) as a point. Which nulls would you reject at the \(5\%\) level? Does the answer change if you use \(\pm 1.96\) instead of the bootstrap bands?

# t-statistic with a jackknife standard error, as above
t_jack <- function(x_hat, null_mean) {
    jack_means <- rep(NA, length(x_hat))
    for (i in seq_along(jack_means)) {
        jack_means[i] <- mean(x_hat[-i])
    }
    jackknife_se <- sd(jack_means) * sqrt(length(x_hat))
    t_hat <- (mean(x_hat) - null_mean) / jackknife_se
    return(t_hat)
}

# One null for each variable, on very different scales
null_means2 <- c(Murder=9, Assault=150, UrbanPop=70)

# Observed t and its bootstrap null distribution for each variable
n_boot <- 999
t2_hats <- rep(NA, length(null_means2))
boot_t_nulls2 <- matrix(NA, nrow=n_boot, ncol=length(null_means2))
for (k in seq_along(null_means2)) {
    x2_k_hat <- USArrests[, names(null_means2)[k]]
    t2_hats[k] <- t_jack(x2_k_hat, null_means2[k])
    for (b in seq(n_boot)) {
        x_boot <- sample(x2_k_hat, replace=TRUE)
        x_boot <- x_boot + (null_means2[k] - mean(x2_k_hat)) # impose the null by recentering
        boot_t_nulls2[b, k] <- t_jack(x_boot, null_means2[k])
    }
}

# Spreadplot: range as a black line, middle 95% as a grey bar, median as a white line
plot(NA, xlim=c(0.5, length(null_means2) + 0.5), ylim=range(boot_t_nulls2, t2_hats),
    xaxt='n', xlab=NA, ylab=expression(hat(t)), main=NA)
axis(1, at=seq_along(null_means2), labels=names(null_means2))
abline(h=0, col=grey(.5), lty=3)                   # zero line
abline(h=c(-1.96, 1.96), col=grey(0.5), lty=3)     # Normal critical values
for (k in seq_along(null_means2)) {
    t_q <- quantile(boot_t_nulls2[, k], probs=c(0, 0.025, 0.5, 0.975, 1))
    segments(k, t_q[1], k, t_q[5], col=rgb(0, 0, 0, .8))
    rect(k - 0.1, t_q[2], k + 0.1, t_q[4], col=grey(0.6), border=NA)
    segments(k - 0.1, t_q[3], k + 0.1, t_q[3], col='white', lwd=2)
}
points(seq_along(null_means2), t2_hats, pch=16, col=rgb(1, 0, 0, .8)) # observed t

# The three means, about 7.8, 170.8, and 65.5, are on very different scales.
# Their observed t-values, about -1.93, 1.73, and -2.14, are not.

Quantiles and Shape Statistics

Bootstrap allows hypothesis tests for any statistic, not just the mean, without relying on parametric theory. For example, the above procedures generalize from means to quantile statistics like medians.

# Test for Median Differences (Impose the Null)
# Bootstrap Null Distribution for the median
# Each Bootstrap shifts medians so that median = null_median

med_hat <- quantile(x_hat, probs=.5)
null_median <- 5.7
boot_medians_null <- rep(NA, 999)
for(b in seq_along(boot_medians_null)){
    x_boot <- sample(x_hat, replace=TRUE) #bootstrap sample
    med_b_hat <- quantile(x_boot, probs=.5) # median
    med_null_b_hat <- med_b_hat + (null_median - med_hat) # impose the null
    boot_medians_null[b] <- med_null_b_hat
}

# 2-Sided Test for Medians
hist(boot_medians_null-null_median, breaks=50,
    border=NA, freq=FALSE, xlab='Null Bootstrap',
    main=NA)
title('Medians (Impose Null)', font.main=1)
median_ci <- quantile(boot_medians_null-null_median, probs=c(.025, .975))
abline(v=median_ci, col=rgb(0, 0, 0, .8), lwd=2)
abline(v=med_hat-null_median, col=rgb(1, 0, 0, .8), lwd=2)

# 2-Sided Test for Median Difference
## Null: No Median Difference
mean(abs(boot_medians_null-null_median) >= abs(med_hat-null_median))
## [1] 0.5995996
# The p-value is about 0.60, so we fail to reject a median of 5.7.
# The population median is about 5.21, so this unrejected null is still false.

TipTest Yourself

Repeat the median test with the murder arrest rates in USArrests, for the hypothesized median of \(7.8\).

x2_hat <- USArrests[, 'Murder'] # murder arrests per 100,000

# Test for Median Differences (Impose the Null)
# Bootstrap Null Distribution for the median
# Each Bootstrap shifts medians so that median = null_median2

med2_hat <- quantile(x2_hat, probs=.5)
null_median2 <- 7.8
boot_medians_null2 <- rep(NA, 999)
for(b in seq_along(boot_medians_null2)){
    x_boot <- sample(x2_hat, replace=TRUE) #bootstrap sample
    med_b_hat <- quantile(x_boot, probs=.5) # median
    med_null_b_hat <- med_b_hat + (null_median2 - med2_hat) # impose the null
    boot_medians_null2[b] <- med_null_b_hat
}

# 2-Sided Test for Medians
hist(boot_medians_null2-null_median2, breaks=50,
    border=NA, freq=FALSE, xlab='Null Bootstrap',
    main=NA)
title('Medians (Impose Null)', font.main=1)
median_ci2 <- quantile(boot_medians_null2-null_median2, probs=c(.025, .975))
abline(v=median_ci2, col=rgb(0, 0, 0, .8), lwd=2)
abline(v=med2_hat-null_median2, col=rgb(1, 0, 0, .8), lwd=2)

# 2-Sided Test for Median Difference
## Null: No Median Difference
mean(abs(boot_medians_null2-null_median2) >= abs(med2_hat-null_median2))
## [1] 0.5395395
# The p-value is about 0.54, so we fail to reject a median of 7.8.

TipTest Yourself

Conduct a hypothesis test for whether the upper quartile is statistically different from \(8\).

q_hat <- quantile(x_hat, probs=.75)

8.4 One-Sided Tests

Above, we tested whether the observed statistic is either extremely high or low. This is known as a two-sided test. There are also two one-sided tests (left tail: observed statistic is extremely low, right tail: observed statistic is extremely high). For example, consider whether the mean statistic, \(M\), is centered on a theoretical value of \(\mathbb{E}[X_{i}]=m_{0}\) for the population. If your null hypothesis is that the theoretical mean is \(m_{0}\), \(H_{0}: \mathbb{E}[X_{i}] = m_{0}\), and you calculated the mean for your sample as \(\hat{m}\), then you can consider any one of these three alternative hypotheses:

• \(H_{A}: \mathbb{E}[X_{i}] \neq m_{0}\), a two-tail test
• \(H_{A}: \mathbb{E}[X_{i}] < m_{0}\), a left-tail test
• \(H_{A}: \mathbb{E}[X_{i}] > m_{0}\), a right-tail test

TipTest Yourself

A fitness tracker manufacturer claims that users take more than \(8{,}000\) steps per day on average. A sample of \(25\) users has \(\hat{m}=8{,}492\) steps and \(\hat{s}_{\text{adj}}=1{,}200\) steps. Test at the \(5\%\) level using theory-based intervals.

(a) Hypotheses. The claim is a right-tail alternative: \(H_{0}: \mathbb{E}[X_{i}] = 8000\) versus \(H_{A}: \mathbb{E}[X_{i}] > 8000\).

(b) Test statistic. The classical standard error is \(\hat{s}_{\text{adj}}/\sqrt{n} = 1200/\sqrt{25} = 240\), so \(t = (8492 - 8000)/240 \approx 2.05\).

(c) Decision. The right-tail critical value at \(5\%\) is \(q_{0.95} \approx 1.645\) from the Normal. Since \(t = 2.05 > 1.645\), we reject \(H_{0}\).

(d) Conclusion. At the \(5\%\) level, the data are consistent with the manufacturer’s claim: there is evidence the population mean exceeds \(8{,}000\) steps per day.

m_steps_hat <- 8492
s_steps_adj_hat <- 1200
n <- 25
steps_null_mean <- 8000
SE <- s_steps_adj_hat / sqrt(n)
t_stat <- (m_steps_hat - steps_null_mean) / SE
t_stat
## [1] 2.05

# Right-tail p-value from the standard Normal approximation
1 - pnorm(t_stat)
## [1] 0.02018222

One-sided hypothesis tests can be conducted by inverting a one-sided confidence interval (covered in the previous chapter) or by computing a one-sided \(p\)-value.

One-Sided \(p\)-values

The \(p\)-value for a one-sided test is more straightforward to implement via a bootstrap null distribution.

For a left-tail test, we examine \[\begin{aligned} p = \text{Prob}( M \leq \hat{m} \mid \mathbb{E}[X_{i}] = m_{0} ) &\approx \text{Prob}( M^{\text{boot}}_{0} \leq \hat{m} ) = \hat{F}^{\text{boot}}_{0}(\hat{m}), \end{aligned}\] where \(\hat{F}^{\text{boot}}_{0}\) is the ECDF of the bootstrap null distribution. We reject the null if \(p \leq 0.05\) at the \(5\%\) level, and otherwise fail to reject.

For a right-tail test, we examine \(p=\text{Prob}( M \geq \hat{m} \mid \mathbb{E}[X_{i}] = m_{0} ) \approx \text{Prob}( M^{\text{boot}}_{0} \geq \hat{m} )\), the share of null bootstrap means at least as large as \(\hat{m}\).

## MAIN EXAMPLE
# Right-tail Test, ALTERNATIVE: mean > 6.5
# Equivalent Visualization with p-value
F_hat <- ecdf(boot_means_null) # Look at right tail
plot(F_hat,
    main=NA,
    xlab='Null Bootstrap')
abline(v=m_hat, col=rgb(1, 0, 0, .8), lwd=2)
p1 <- mean(boot_means_null >= m_hat) #Compute right tail prob: 0.98
title( paste0('p=', round(p1, 3)), font.main=1)

if(p1 >.05){
    print('fail to reject the null that mean=6.5, at the 5% level')
} else {
    print('reject the null that mean=6.5 in favor of >6.5, at the 5% level')
}
## [1] "fail to reject the null that mean=6.5, at the 5% level"
# The sample mean is below 6.5, so it gives no evidence for a mean above 6.5.

The left-tail test uses the other side of the same null distribution.

# Left-tail Test, ALTERNATIVE: mean < 6.5
# Compute left-tail prob: p = F(m_hat)
p_left <- F_hat(m_hat)
p_left
## [1] 0.01901902

if(p_left >.05){
    print('fail to reject the null that mean=6.5, at the 5% level')
} else {
    print('reject the null that mean=6.5 in favor of <6.5, at the 5% level')
}
## [1] "reject the null that mean=6.5 in favor of <6.5, at the 5% level"
# The left-tail p-value is about 0.02, so we favor a mean below 6.5.

TipTest Yourself

Repeat both one-sided tests with the murder arrest rates, using boot_means_null2 from the Test Yourself in Impose the Null.

# Right-tail Test, ALTERNATIVE: mean > 9
# Equivalent Visualization with p-value
F2_hat <- ecdf(boot_means_null2) # Look at right tail
plot(F2_hat,
    main=NA,
    xlab='Null Bootstrap')
abline(v=m2_hat, col=rgb(1, 0, 0, .8), lwd=2)
p1_2 <- mean(boot_means_null2 >= m2_hat) #Compute right tail prob
title( paste0('p=', round(p1_2, 3)), font.main=1)

if(p1_2 >.05){
    print('fail to reject the null that mean=9, at the 5% level')
} else {
    print('reject the null that mean=9 in favor of >9, at the 5% level')
}
## [1] "fail to reject the null that mean=9, at the 5% level"
# The sample mean is below 9, so it gives no evidence for a mean above 9.

# Left-tail Test, ALTERNATIVE: mean < 9
# Compute left-tail prob: p = F(m_hat)
p_left2 <- F2_hat(m2_hat)
p_left2
## [1] 0.02102102

if(p_left2 >.05){
    print('fail to reject the null that mean=9, at the 5% level')
} else {
    print('reject the null that mean=9 in favor of <9, at the 5% level')
}
## [1] "reject the null that mean=9 in favor of <9, at the 5% level"
# The left-tail p-value is about 0.02, so we favor a mean below 9.

TipTest Yourself

For two-sided tests, measure distance from \(m_{0}\) by subtracting it before taking absolute values. For one-sided tests, subtracting \(m_{0}\) from both the null-bootstrap statistic and the observed statistic leaves their ordering unchanged. Specifically, \(p = \text{Prob}( M \leq \hat{m} \mid \mathbb{E}[X_{i}] = m_{0} ) = \text{Prob}( M - m_{0} \leq \hat{m} - m_{0} \mid \mathbb{E}[X_{i}] = m_{0} )\). That is intuitively also why the \(t\)-value can be used for both one and two-sided hypothesis tests.

# See that the 'recentering' matters for two-sided tests
ecdf( abs(boot_means_null-null_mean) )( abs(m_hat-null_mean) )
## [1] 0.960961
ecdf( abs(boot_means_null) )( abs(m_hat) )
## [1] 0.01901902

# See that the 'recentering' doesn't matter for one-sided ones
ecdf( boot_means_null-null_mean)( m_hat-null_mean)
## [1] 0.01901902
ecdf( boot_means_null )( m_hat)
## [1] 0.01901902

8.5 Exercises

1. Comment the script you wrote for this chapter, then restart R and check that it runs from a clean session, then check the script with AI as explained in Working with AI. Write three sentences from memory on the main statistical idea of this chapter, and ask the assistant what is wrong, vague, or missing. Finish with your own questions about whatever you found hardest.

2. Explain the difference between “rejecting the null hypothesis” and “proving the alternative hypothesis is true.” Why is the phrase “fail to reject” used instead of “accept the null”?

3. Suppose you collect a sample of \(n = 50\) observations with \(\hat{m} = 22.4\) and \(\hat{s}_{\text{adj}} = 6.0\). You want to test \(H_{0}: \mathbb{E}[X_{i}] = 20\) against \(H_{A}: \mathbb{E}[X_{i}] \neq 20\) at the \(5\%\) level. Compute the \(t\)-value using theory-driven standard errors and determine whether you reject or fail to reject.

4. Using the USArrests dataset in R, test the hypothesis that the population mean of Assault equals \(150\) by constructing a bootstrap null distribution. Compute the two-sided \(p\)-value and state whether you reject or fail to reject at the \(5\%\) level.

Further Reading

• https://learningstatisticswithr.com/book/hypothesistesting.html – Comprehensive chapter on null hypotheses, p-values, and test decisions with R examples.
• https://www.khanacademy.org/math/statistics-probability/significance-tests-one-sample – Step-by-step video lessons on hypothesis testing for one-sample means and proportions.

Recall

Besides inverting a CI, we built hypothesis tests by imposing the null. A smaller level \(\alpha\) means higher coverage, a wider interval, and fewer rejections. A null hypothesis comes from outside the sample, and failing to reject it does not show that it is true. We also summarized the evidence with the \(p\)-value and the \(t\)-value. “Statistically significant” says only that the \(p\)-value fell below a chosen cutoff. Even when the null is true, about \(5\%\) of samples give a \(p\)-value below \(0.05\). Running many tests inflates the chance of a false rejection. The fitness-tracker example tied these ideas together: with \(\hat{m}=8492\), \(\hat{s}_{\text{adj}}=1200\), and \(n=25\), the standard error is \(\text{SE} = 1200/\sqrt{25} = 240\) and the \(t\)-value for \(H_{0}: \mathbb{E}[X_{i}] = 8000\) is \((8492 - 8000)/240 \approx 2.05\), which exceeds the right-tail critical value \(1.645\) at the \(5\%\) level and so rejects the null.

Abadie, Alberto. 2020. “Statistical Nonsignificance in Empirical Economics.” American Economic Review: Insights 2 (2): 193–208. https://doi.org/10.1257/aeri.20190252.

Adamson, Jordan. 2022. “An Econometric Model of Ethnic Maps and Evaluation of the Endogenous Areal Unit Problem.” Working Paper. In preparation. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4091833.

Adamson, Jordan. 2025. “Trade and the Rise of Ancient Greek City-States.” Journal of Economic Behavior & Organization 235: 107035. https://doi.org/10.1016/j.jebo.2025.107035.

Adamson, Jordan, and Erik O. Kimbrough. 2023. “The Supply Side Determinants of Territory.” Journal of Peace Research 60 (2): 209–25. https://doi.org/10.1177/00223433221075006.

Adamson, Jordan, and Bart J. Wilson. 2024. “Territory in the State of Nature.” Journal of Economic Behavior & Organization 219: 490–509. https://doi.org/10.1016/j.jebo.2024.01.032.

Altman, Douglas G., and J. Martin Bland. 1995. “Statistics Notes: Absence of Evidence Is Not Evidence of Absence.” BMJ 311 (7003): 485. https://doi.org/10.1136/bmj.311.7003.485.

Baddeley, Adrian, Ege Rubak, and Rolf Turner. 2015. Spatial Point Patterns: Methodology and Applications with r. Chapman; Hall/CRC. https://doi.org/10.1201/b19708.

Besag, Julian, and Peter J. Diggle. 1977. “Simple Monte Carlo Tests for Spatial Pattern.” Applied Statistics 26 (3): 327–33. https://doi.org/10.2307/2346974.

Bonferroni, Carlo E. 1936. “Teoria statistica delle classi e calcolo delle probabilità.” Pubblicazioni del R. Istituto Superiore di Scienze Economiche e Commerciali di Firenze 8: 3–62.

Davison, A. C., and D. V. Hinkley. 1997. Bootstrap Methods and Their Application. Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press.

Hope, Adery C. A. 1968. “A Simplified Monte Carlo Significance Test Procedure.” Journal of the Royal Statistical Society: Series B (Methodological) 30 (3): 582–98. https://doi.org/10.1111/j.2517-6161.1968.tb00759.x.

Imbens, Guido W. 2021. “Statistical Significance, p-Values, and the Reporting of Uncertainty.” Journal of Economic Perspectives 35 (3): 157–74. https://doi.org/10.1257/jep.35.3.157.

Lehmann, E. L., and Joseph P. Romano. 2005. Testing Statistical Hypotheses. 3rd ed. Springer Texts in Statistics. Springer. https://doi.org/10.1007/0-387-27605-X.

McCloskey, Deirdre N., and Stephen T. Ziliak. 1996. “The Standard Error of Regressions.” Journal of Economic Literature 34 (1): 97–114. http://www.jstor.org/stable/2729411.

Neyman, Jerzy, and Egon S. Pearson. 1933. “On the Problem of the Most Efficient Tests of Statistical Hypotheses.” Philosophical Transactions of the Royal Society of London. Series A 231: 289–337. https://doi.org/10.1098/rsta.1933.0009.

Wasserstein, Ronald L., and Nicole A. Lazar. 2016. “The ASA Statement on p-Values: Context, Process, and Purpose.” The American Statistician 70 (2): 129–33. https://doi.org/10.1080/00031305.2016.1154108.