Unit6 Parametric Point Estimation

資料來源：Text book: 1. Introduction to the Theory of Statistics by Mood Graybill et. al. 2. Introduction to Mathematical Statistics by Robert V. Hogg & Allen T. Craig

The main methods of finding estimators: 1. method of moments 2. method of maximum likelihood Definition: A point estimator is any function T ( X 1 , L, X n ) of a sample. That is, any statistic is a point estimator.

資料來源：Text book: 1. Introduction to the Theory of Statistics by Mood Graybill et. al. 2. Introduction to Mathematical Statistics by Robert V. Hogg & Allen T. Craig

Method of moments: Let X 1 , L, X n be a sample from a population with p.d.f. or p.m.f. f ( x | θ1 , L, θ k ). The method of moments estimators are found by equating the first k sample moments to the corresponding k population moments. That is 1 n X i = E( X ) ∑ n i =1 1 n 2 2 X = E ( X ) ∑ i n i =1 M 1 n k k X = E ( X ) ∑ i n i =1 資料來源：Text book: 1. Introduction to the Theory of Statistics by Mood Graybill et. al. 2. Introduction to Mathematical Statistics by Robert V. Hogg & Allen T. Craig

2 N ( θ , σ ). Find the X , L , X Example: Let 1 n are i.i.d. 2 σ θ method of moments estimators of and .

Example: Let X 1 , L , X n are i.i.d. U (0, θ ). Find the method of moments estimator of θ .

資料來源：Text book: 1. Introduction to the Theory of Statistics by Mood Graybill et. al. 2. Introduction to Mathematical Statistics by Robert V. Hogg & Allen T. Craig

Remark: If X 1 , L, X n ~ f ( x | θ ), and θˆ is the method of moments estimator of θ and g (θ ) is a function of θ , then g (θˆ) is the method of moments estimator of g (θ ). i .i .d .

Example: Let X 1 , L, X n ~ poisson (θ ). Find the method of moments estimator of τ (θ ) = e −θ .

資料來源：Text book: 1. Introduction to the Theory of Statistics by Mood Graybill et. al. 2. Introduction to Mathematical Statistics by Robert V. Hogg & Allen T. Craig

Definition: The likelihood function of n random variables X 1 , L , X n is defined to be the joint density of the n random variables, which is considered to be a function of unknown parameters. The likelihood function is denoted by L(θ ) = f X 1 , L, X n ( x1 , L x n ; θ )

資料來源：Text book: 1. Introduction to the Theory of Statistics by Mood Graybill et. al. 2. Introduction to Mathematical Statistics by Robert V. Hogg & Allen T. Craig

Example: Find the likelihood function if X 1 , L , X n are i.i.d. Bernoulli (θ ). Example: Find the likelihood function if X 1 , L , X n are N (θ , σ 2 ). Example: Find the likelihood function if X 1 , L , X n are U (0, θ ).

資料來源：Text book: 1. Introduction to the Theory of Statistics by Mood Graybill et. al. 2. Introduction to Mathematical Statistics by Robert V. Hogg & Allen T. Craig

Definition: If θˆ maximizes L(θ ), then θˆ( X 1 , L, X n ) is the maximum likelihood estimator of θ , θˆMLE . Remark: L(θ ) and lnL(θ ) have their maxima at the same value of θ .

資料來源：Text book: 1. Introduction to the Theory of Statistics by Mood Graybill et. al. 2. Introduction to Mathematical Statistics by Robert V. Hogg & Allen T. Craig

i .i .d .

2 ˆ ˆ σ θ Example: Let X 1 , L, X n ~ N (θ , σ ). Find MLE and MLE .

2

i .i .d .

Example: Let X 1 , L, X n ~ Bernoulli (θ ). Find θˆMLE . i .i .d .

Example: Let X 1 , L, X n ~ U (0, θ ). Find θˆMLE . i .i .d .

Example: Let X 1 , L, X n ~ f ( x) = e −( x −θ ) ⋅ I [θ , ∞ ) ( x). Find θˆMLE .

資料來源：Text book: 1. Introduction to the Theory of Statistics by Mood Graybill et. al. 2. Introduction to Mathematical Statistics by Robert V. Hogg & Allen T. Craig

Theorem ( Invariance property of Maximum likelihood estimators ): If θˆ is the MLE of θ , then for any function τ (θ ), the MLE of τ (θ ) is τ (θˆ). i .i .d .

Example: Let X 1 , L, X n ~ Poisson (θ ). Find the MLE −λ ( ) ( 1 ). τ λ = λ + e of

資料來源：Text book: 1. Introduction to the Theory of Statistics by Mood Graybill et. al. 2. Introduction to Mathematical Statistics by Robert V. Hogg & Allen T. Craig

The ways to evaluate the estimators: 1. closeness (a) more concentrated or most concentrated (b) pitman-closer or pitman-closet 2. mean squared error 3. unbiased 4. consistency 資料來源：Text book: 1. Introduction to the Theory of Statistics by Mood Graybill et. al. 2. Introduction to Mathematical Statistics by Robert V. Hogg & Allen T. Craig

' ' τ ( θ ). T T T Definition: Let and be two estimators of

is called a more concentrated estimator of τ (θ ) then T iff Pr (| T ' − τ (θ ) |< λ ) ≥ Pr (| T − τ (θ ) |< λ ), ∀λ > 0 and ∀θ ∈ Θ. An estimator T * is called most concentrated if it is more concentrated than any other estimator.

2 Example: If X 1 ~ N (θ , σ 12 ) and X 2 ~ N (θ , σ 2 ), prove that X 1 is more concentrated about θ then X 2

2 2 σ ≤ σ iff 1 2.

資料來源：Text book: 1. Introduction to the Theory of Statistics by Mood Graybill et. al. 2. Introduction to Mathematical Statistics by Robert V. Hogg & Allen T. Craig

Definition: Let T and T ' be two estimators of τ (θ ). T ' is called a Pitman-closer estimator of τ (θ ) than T iff Pr[| T ' − τ (θ ) 0 and ∀θ ∈ Θ, lim Pr (| Wn − θ |< ε ) = 1. n →∞

Remark: If an estimator is a mean-squared-error consistent estimator, then it is also a simple consistent estimator.

資料來源：Text book: 1. Introduction to the Theory of Statistics by Mood Graybill et. al. 2. Introduction to Mathematical Statistics by Robert V. Hogg & Allen T. Craig

Definition: Let X 1 , L , X n denote a random sample that has p.d.f. f ( x; θ ). A real valued function S = S ( X 1 , L , X n ) is called sufficient statistic for θ iff the conditional distribution of X 1 , L , X n given S = s is free from θ . i.e.

f ( x1 ; θ ) L f ( xn ; θ ) = H ( x1 , L , xn ). g s ( s; θ )

資料來源：Text book: 1. Introduction to the Theory of Statistics by Mood Graybill et. al. 2. Introduction to Mathematical Statistics by Robert V. Hogg & Allen T. Craig

i .i .d .

Example: Let X 1 , Ln , X n ~ Bernoulli(θ ), 0 < θ < 1. Show that S = ∑ X i is a sufficient statistic for θ . i =1

i .i .d .

2 Example: Let X 1 , L , X n ~n N (θ , σ ),where σ 2 is known. Show that S = ∑ X i is sufficient statistic for θ . i =1

i .i .d .

Example: Let X 1 , L , X n ~ f ( x | θ ) = e −( x −θ ) , 0 < x < ∞, − ∞ < θ < ∞. Show that Y1 = min( X 1 , L , X n ) is sufficient statistic for θ .

資料來源：Text book: 1. Introduction to the Theory of Statistics by Mood Graybill et. al. 2. Introduction to Mathematical Statistics by Robert V. Hogg & Allen T. Craig

i .i .d .

Factorization Theorem: Let X 1 , L , X n be ~ f ( x | θ ). The statistic S ( X 1 , L , X n ) is sufficient for θ iff we can write the joint density/ pmf f ( x1 , L , xn | θ ) as f ( x1 , L , xn | θ ) = h( x1 , L , xn ) g ( s( x1 , L, xn ) | θ ) for some suitable function S ( X 1 , L , X n ).

資料來源：Text book: 1. Introduction to the Theory of Statistics by Mood Graybill et. al. 2. Introduction to Mathematical Statistics by Robert V. Hogg & Allen T. Craig

i .i .d .

Example: Let X 1 , L, X n ~ poisson(λ ). Find a sufficient statistic of λ. i .i .d .

Example: Let X 1 , L , X n ~ f ( x | θ ) = θe −θx . Find a sufficient statistic for θ . i .i .d .

2x

Example: Let X 1 , L , X n ~ f ( x | θ ) = 2 I ( 0, θ ) ( x). θ sufficient statistic for θ .

Find a

資料來源：Text book: 1. Introduction to the Theory of Statistics by Mood Graybill et. al. 2. Introduction to Mathematical Statistics by Robert V. Hogg & Allen T. Craig

Theorem: Let X 1 , L , X n denote a random sample from a distribution that has p.d.f. f ( x; θ ). If a sufficient statistic S for θ exists and θˆ is a maximum likelihood estimator of θ , then θˆ is a function of S. Theorem: If S is a sufficient statistic, then any one-to-one functions, or transformations, of S is also sufficient.

資料來源：Text book: 1. Introduction to the Theory of Statistics by Mood Graybill et. al. 2. Introduction to Mathematical Statistics by Robert V. Hogg & Allen T. Craig

Definition: A family of p.d.f.s or p.m.f.s is called an exponential family if it can be expressed as f ( x | θ ) = a (θ )b( x) exp[c(θ )d ( x)]. Here b( x) ≥ 0 and d (x) are real-valued functions of the observation x, and a (θ ) ≥ 0 and c(θ ) are real-valued functions of the possibly values of the parameter θ .

資料來源：Text book: 1. Introduction to the Theory of Statistics by Mood Graybill et. al. 2. Introduction to Mathematical Statistics by Robert V. Hogg & Allen T. Craig

Example: Let f ( x; θ ) ~ N (θ , σ 2 ), where σ 2 is known. Does f ( x;θ ) belong to an exponential family? Example: Let f ( x; p) ~ B(n, p), where n is known. Does f ( x;θ ) belong to an exponential family?

資料來源：Text book: 1. Introduction to the Theory of Statistics by Mood Graybill et. al. 2. Introduction to Mathematical Statistics by Robert V. Hogg & Allen T. Craig

n

Theorem: If f ( x | θ ) = a(θ )b( x) exp[c(θ )d ( x)], then ∑ d ( X i ) i =1 is a sufficient statistic. i .i .d .

2 Example: Let X 1 , L , X n ~ N (θ , σ ), where σ 2 is known. Find a sufficient statistic of θ .

i .i .d .

Example: Let X 1 , L, X n ~ B(n, p), where n is known. Find a sufficient statistic of p.

資料來源：Text book: 1. Introduction to the Theory of Statistics by Mood Graybill et. al. 2. Introduction to Mathematical Statistics by Robert V. Hogg & Allen T. Craig

Definition: An estimator T * is a best unbiased estimator * E ( T ) = τ (θ ) for all θ of τ (θ ) if it satisfies and, for any other estimator T with E (T ) = τ (θ ), we have Var (T * ) ≤ Var (T ) for all θ . T * is also called a uniform minimum variance unbiased estimator (UMVUE) of τ (θ ).

資料來源：Text book: 1. Introduction to the Theory of Statistics by Mood Graybill et. al. 2. Introduction to Mathematical Statistics by Robert V. Hogg & Allen T. Craig

Rao-Blackwell Theorem: Let T be any unbiased estimator of τ (θ ), and let S be a sufficient statistic for θ . Define T ' = E (T | S ). Then E (T ' ) = τ (θ ) and Var (T ' ) ≤ Var (T ), ∀θ , that is, T ' is a uniformly better unbiased estimator of τ (θ ).

資料來源：Text book: 1. Introduction to the Theory of Statistics by Mood Graybill et. al. 2. Introduction to Mathematical Statistics by Robert V. Hogg & Allen T. Craig

i .i .d .

Example: Let X 1 , L, X n ~ Bernoulli (θ ). Find a better unbiased estimator of θ than X 1 . i .i .d .

Example: Let X 1 , L, X n ~ Poisson (λ ). Find a better −λ τ ( λ ) = λ e unbiased estimator of than I {1} ( x1 ). i .i .d .

Example: Let X 1 , L, X n ~ B(k , θ ), where k is known. Find a better unbiased estimator of τ (θ ) = kθ (1 − θ ) k −1 than I {1} ( x1 ).

資料來源：Text book: 1. Introduction to the Theory of Statistics by Mood Graybill et. al. 2. Introduction to Mathematical Statistics by Robert V. Hogg & Allen T. Craig

Theorem: (Cramer-Rao inequality): Let X 1 , L, X n be i.i.d. with p.d.f. f ( x | θ ), and let T ( x~ ) = T ( X 1 , L, X n ) be any unbiased estimator of τ (θ ) where E[T ( x)] is a differentiable ~ function of θ . Suppose the joint p.d.f. f ( x~ | θ ) = f ( x1 , L, xn | θ ) satisfies d ∂ L ∫ h( x) f ( x | θ )dx1 L dxn = ∫ L ∫ h( x) f ( x | θ )dx1 L dxn , ∫ ~ ~ ~ ∂θ ~ dθ

for any function h( x) with E | h( x) |< ∞. Then ~

Var[T ( x)] ≥ ~

[τ ' (θ )]2

~

∂ nE[( log f ( x | θ )) 2 ] ∂θ

.

Equality holds iff there exist a function, say k (θ , n) n ∂ such that ∑ log f ( x | θ ) = k (θ , n)[T ( x1 , L, xn ) − τ (θ )]. i =1 ∂θ 資料來源：Text book: 1. Introduction to the Theory of Statistics by Mood Graybill et. al. 2. Introduction to Mathematical Statistics by Robert V. Hogg & Allen T. Craig

i .i .d .

X 1 , L , X n ~ Bernoulli(θ ). Example: Let Find the Cramer-Rao lower bound for the variance of unbiased estimator of θ . i .i .d .

Example: Let X 1 , L , X n ~ N (θ , 1). Find the CramerRao lower bound for the variance of unbiased estimator of θ .

資料來源：Text book: 1. Introduction to the Theory of Statistics by Mood Graybill et. al. 2. Introduction to Mathematical Statistics by Robert V. Hogg & Allen T. Craig

Remark: 1. If an unbiased estimator whose variance is the same as the Cramer-Rao lower bound, then this estimator is the UMVUE. 2. If there exists an unbiased estimator T = T ( X 1 , L , X n ) of τ (θ ) such that ∂ log f ( x; θ ) = k (θ , n)[T − τ (θ )]. ∑ i =1 ∂θ n

for some functions k (θ , n) and τ (θ ), then T is the UMVUE of τ (θ ).

資料來源：Text book: 1. Introduction to the Theory of Statistics by Mood Graybill et. al. 2. Introduction to Mathematical Statistics by Robert V. Hogg & Allen T. Craig

3. An UMVUE can exist whose variance does not coincide with the Cramer-Rao lower bound. 4. We will be able to find an estimator whose variance is the same as Cramer-Rao lower bound iff the density from which we are sampling is a member of the exponential class.

資料來源：Text book: 1. Introduction to the Theory of Statistics by Mood Graybill et. al. 2. Introduction to Mathematical Statistics by Robert V. Hogg & Allen T. Craig

Example: Let X 1 , L , X n be i.i.d. Bernoulli (θ ), 0 < θ < 1. Is these a function of θ for which there exists an unbiased estimator whose variance coincides with the Cramer-Rao lower bound? log θ x θ I ( 0, 1) ( x), Example: Let X 1 , L , X n be i.i.d. f ( x; θ ) = θ −1 θ > 1. Find a function of θ for which there

exists an unbiased estimator whose variance coincides with the Cramer-Rao lower bound.

資料來源：Text book: 1. Introduction to the Theory of Statistics by Mood Graybill et. al. 2. Introduction to Mathematical Statistics by Robert V. Hogg & Allen T. Craig

Definition: Let f (t | θ ) be a family of p.d.f.s. or p.m.f.s. for a statistic T ( x~ ). The family of probability dist ri buti ons is called complete iff E [g (T )] = 0, ∀θ implies Pr[ g (T ) = 0] = 1, ∀θ Equivalently, T ( x~ ). is called a complete statistic. Example: Let T have a B (n, θ ) distribution, 0 < θ < 1. Is T a complete statistic?

資料來源：Text book: 1. Introduction to the Theory of Statistics by Mood Graybill et. al. 2. Introduction to Mathematical Statistics by Robert V. Hogg & Allen T. Craig

Definition: A sufficient statistic S ( x) is called a minimal ~ sufficient statistic if, for any other sufficient ' ' S ( x ), S ( x ) S ( x). statistic is a function of ~ ~ ~

i .i .d .

Theorem: Let X 1 , L, X n ~ f ( x; θ ). If n f ( x; θ ) = a (θ )b( x) exp[c(θ )d ( x)], then∑i =1 d ( xi ) is a complete minimal sufficient statistic.

資料來源：Text book: 1. Introduction to the Theory of Statistics by Mood Graybill et. al. 2. Introduction to Mathematical Statistics by Robert V. Hogg & Allen T. Craig

Lehmann-Scheffe Theorem: Let X 1 , L , X n be a random sample from a density f ( x; θ ). If S is a complete sufficient statistic and T = t (s ), a function of S , is an unbiased estimator of τ (θ ), then T is the UMVUE of τ (θ ).

資料來源：Text book: 1. Introduction to the Theory of Statistics by Mood Graybill et. al. 2. Introduction to Mathematical Statistics by Robert V. Hogg & Allen T. Craig

i .i .d .

Example: Let X 1 , L, X n ~ Bernoulli (θ ). Find the UMVUE of θ . i .i .d .

Example: Let X 1 , L, X n ~ Poisson (λ ). Find the −λ e . UMVUE of i .i .d .

Example: Let X 1 , L, X n ~ f ( x;θ ) = θe −θx ⋅ I ( 0, ∞ ) ( x). Find the UMVUE of Var ( X ).

資料來源：Text book: 1. Introduction to the Theory of Statistics by Mood Graybill et. al. 2. Introduction to Mathematical Statistics by Robert V. Hogg & Allen T. Craig