General Formula

Click here to expand...

CI Definition

An interval [𝐴, 𝐵] is a (1 − 𝛼)100% confidence interval for the parameter 𝜃 if it contains the parameter with probability (1 − 𝛼):

• 𝐏{𝐴 ≤ 𝜃 ≤ 𝐵} = 1 − 𝛼

where:

• 𝛼 - significance level
• (1 − 𝛼) - confidence level or coverage probability

CI Formula Intuition

Click here to expand...

Given a sample of data and a desired confidence level (1 − 𝛼), how can we construct a confidence interval [𝐴, 𝐵] that will satisfy the coverage condition

• 𝐏{𝐴 ≤ 𝜃 ≤ 𝐵} = 1 − 𝛼

first we need to estimate 𝜃

choose an unbiased estimator with normal distribution (e.g. MLE)

use the estimator to take the sample data and estimate 𝜃 point estimate 𝜃ˆ

next we standardize 𝜃ˆ to get a standard normal variable 𝑧:

• 𝑧 = [𝜃ˆ - 𝐄(𝜃ˆ)] / 𝜎(𝜃ˆ)

since 𝜃ˆ was estimated with an unbiased estimator: 𝐄(𝜃ˆ) = 𝜃

• 𝑧 = (𝜃ˆ - 𝜃) / 𝜎(𝜃ˆ)

this variable 𝑧 falls between the standard normal quantiles 𝑞_𝛼/2and 𝑞_1−𝛼/2, denoted by

• -𝑧_𝛼/2= 𝑞_𝛼/2
• 𝑧_𝛼/2= 𝑞_1−𝛼/2

with probability (1 - 𝛼), then:

• 𝐏{-𝑧_𝛼/2 ≤ (𝜃ˆ - 𝜃) / 𝜎(𝜃ˆ) ≤ 𝑧_𝛼/2} = 1 - 𝛼

now rearrange for 𝜃:

• 𝐏{𝜃ˆ - 𝑧_𝛼/2·𝜎(𝜃ˆ) ≤ 𝜃 ≤ 𝜃ˆ + 𝑧_𝛼/2·𝜎(𝜃ˆ)} = 1 - 𝛼

we have obtained two numbers:

• 𝐴 = 𝜃ˆ - 𝑧_𝛼/2·𝜎(𝜃ˆ)
• 𝐵 = 𝜃ˆ + 𝑧_𝛼/2·𝜎(𝜃ˆ)

such that

• 𝐏{𝐴 ≤ 𝜃 ≤ 𝐵} = 1 − 𝛼

CI Formulas

Large Sample Size (𝑛)	Normal Population	𝑆𝐸(𝜃ˆ) Known	Confidence Interval
FALSE	FALSE	EITHER	Bootstrap Method
FALSE	TRUE	FALSE	𝜃ˆ ± 𝑡𝛼/2·𝑆𝐸ˆ(𝜃ˆ)
FALSE	TRUE	TRUE	𝜃ˆ ± 𝑧𝛼/2·𝑆𝐸(𝜃ˆ)
TRUE	EITHER	FALSE	𝜃ˆ ± 𝑧𝛼/2·𝑆𝐸ˆ(𝜃ˆ)
TRUE	EITHER	TRUE	𝜃ˆ ± 𝑧𝛼/2·𝑆𝐸(𝜃ˆ)

where:

• 𝜃ˆ - point estimate/statistic or center of the interval
• 𝑧 - z-score a type of confidence multiplier
• 𝑡 - t-score a type of confidence multiplier
• 𝑆𝐸(𝜃ˆ) or 𝜎(𝜃ˆ) or 𝑆𝑡𝑑(𝜃ˆ) - standard error of the point estimator/statistic
• 𝑆𝐸ˆ(𝜃ˆ) or 𝑠(𝜃ˆ) or 𝑆𝑡𝑑ˆ(𝜃ˆ) - estimated standard error of the point estimator/statistic

CI Annotated

CI Diagram

Link to original

CI for Sample Mean - When is 𝜎 Known

Click here to expand...

the general formula states the confidence interval is:

• 𝜃ˆ ± 𝑧*·𝑆𝐸(𝜃ˆ)

computing CI for population mean, we substitute:

• 𝜃ˆ = 𝑋̅
• 𝑆𝐸(𝜃ˆ) = 𝜎/√𝑛

therefore:

where:

• 𝑋̅ is the point estimate/statistic (in this case, the sample mean notation was used)
• 𝑧* is the z-score which is a type of confidence multiplier computed from the z-distribution
• 𝜎 is the true population standard deviation
• 𝑛 is sample size
• 𝜎/√𝑛 - is the standard error of sample mean
• 𝑧*·(𝜎/√𝑛) is called the margin of error

The confidence interval is therefore centered at the estimate and its width is exactly (2 * margin of error)

CI for Sample Mean - When 𝜎 is Unknown & 𝑛 is Small

Click here to expand...

replace population standard deviation (𝜎) with sample standard deviation (𝑠)

• 𝑋̅ ± 𝑡*·(𝑠/√𝑛)

where:

• 𝑡* - is the confidence multiplier, t-distribution
• 𝑠 - is the sample standard deviation
• 𝑠/√𝑛 - is the estimated standard error of point estimate

The bad news is that once 𝜎 (sigma) has been replaced by 𝑠, we lose the Central Limit Theorem (CLT), together with the normality of 𝑋̅, and therefore the confidence multipliers 𝑧* for the different levels of confidence (1.645, 1.96, 2.576) are (generally) not correct any more. The new multipliers come from a different distribution called the “t distribution” and are therefore denoted by 𝑡*

CI for Sample Mean - When 𝜎 is Unknown & 𝑛 is Large

Click here to expand...

It turns out that for large values of 𝑛, the 𝑡* multipliers are not that different from the 𝑧* multipliers, and therefore using the interval formula

• 𝑋̅ ± 𝑧*·(𝑠/√𝑛)

for 𝜇 (mu) when 𝜎 (sigma) is unknown provides a pretty good approximation

CIs for Sample Mean