Bayesian intrinsic point estimation Teng Zhang

Fan Wu

Department of Statistics North Carolina State University

December 5, 2012

ncsu-red

Bayesian intrinsic point estimation

1 / 17

c 2012 by Teng Zhang Fan Wu

Intrinsic estimation

Bayes estimator In Bayesian point estimation, loss function ˜ (θ, λ)} denotes the loss incurred when the L{θ, ˜ parameter is (θ, λ) and the action taken is θ. Given data x, the Bayes estimator θ∗ (x) is the value in Θ that minimizes the posterior expected loss ˜ L{θ|x}, i.e. ˜ θ∗ (x) = arg min L{θ|x} ˜ θ∈Θ Z Z ˜ (θ, λ)}f (θ, λ|x)dθdλ. L{θ, = arg min ˜ θ∈Θ

ncsu-red

Λ

Θ

Bayesian intrinsic point estimation

2 / 17

c 2012 by Teng Zhang Fan Wu

Intrinsic estimation

Motivation of intrinsic estimation The loss function has various kinds of form, for example, squared loss, ˜ θ} = (θ˜ − θ)2 . L{θ, Under squared loss, the Bayes estimator is not invariant under one-to-one transformations of the data or the parameter space. However, intrinsic loss functions, which shifts attention from the discrepancy between the estimate θ˜ and the true value θ, to the more relevant discrepancy between the statistical models they label. ncsu-red

Bayesian intrinsic point estimation

3 / 17

c 2012 by Teng Zhang Fan Wu

Intrinsic estimation

Kullback-Leibler divergence Suppose p and q are two probability distribution functions. The Kullback-Leibler divergence [4] of p and q is defined as Z ∞ p(x) κ(p|q) = dx. p(x) log q(x) −∞ The Kullback-Leibler divergence of two probability distributions p and q is non-negative, equal to zero if and only if p(x) = q(x) almost everywhere. But it is not symmetric. ncsu-red

Bayesian intrinsic point estimation

4 / 17

c 2012 by Teng Zhang Fan Wu

Intrinsic estimation

Intrinsic discrepancy Suppose p is the probability distribution function of X ,the intrinsic discrepancy [2] [3] between two models p(x|θ1 ) and p(x|θ2 ) for data x ∈ χ is defined by δ{θ1 , θ2 } = min{κ(θ1 |θ2 ), κ(θ2 |θ1 )}, where Z κ(θi |θj ) =

p(x|θj ) log χ

p(x|θj ) dx. p(x|θi )

The intrinsic discrepancy is symmetric. ncsu-red

Bayesian intrinsic point estimation

5 / 17

c 2012 by Teng Zhang Fan Wu

Intrinsic estimation

Intrinsic loss function Suppose px (·|θ, λ) is the probability distribution function of X , then the intrinsic loss function [1] of parameter θ is ˜ λ), ˜ (θ, λ)}, ˜ (θ, λ)} = inf δ{(θ, L{θ, ˜ λ∈Λ

where the actual parameter values are (θ, λ).

ncsu-red

Bayesian intrinsic point estimation

6 / 17

c 2012 by Teng Zhang Fan Wu

Intrinsic estimation

Computation of the intrinsic discrepancy loss Let F be a parametric family of probability distributions F = {p(x|θ, λ), θ ∈ Θ, λ ∈ Λ, x ∈ χ(θ, λ)}, with convex support χ(θ, λ) for all θ and λ. Then, [1] ˜ (θ, λ)} = inf min{κ{θ, ˜ λ|θ, ˜ λ}, κ{θ, λ|θ, ˜ λ}} ˜ L{θ, ˜ λ∈Λ

˜ λ|θ, ˜ λ}, inf κ{θ, λ|θ, ˜ λ}}. ˜ = min{ inf κ{θ, ˜ λ∈Λ

ncsu-red

˜ λ∈Λ

Bayesian intrinsic point estimation

7 / 17

c 2012 by Teng Zhang Fan Wu

Intrinsic estimation

Bayesian intrinsic estimator The intrinsic discrepancy loss is given by Z Z ˜ = ˜ (θ, λ)}f (θ, λ|x)dθdλ, d(θ|x) L{θ, Θ

Ω

and the intrinsic estimator of θ is the solution which ˜ minimizes d(θ|x).

ncsu-red

Bayesian intrinsic point estimation

8 / 17

c 2012 by Teng Zhang Fan Wu

Computation details

Intrinsic loss functions Suppose x1 , · · · , xn are i.i.d from N(µ, σ 2 ). By [1], the loss function of mean µ and variance σ 2 are n (˜ µ − µ)2 L(˜ µ, (µ, σ )) = log(1 + ), 2 σ2  n g(θ) L(˜ σ 2 , (µ, σ 2 )) = n2 . g(1/θ) 2 2

where g(t) = (t − 1) − log(t), and θ = σ ˜ 2 /σ 2 .

ncsu-red

Bayesian intrinsic point estimation

9 / 17

c 2012 by Teng Zhang Fan Wu

Computation details

Prior selected We use Jeffrey’s prior of µ and σ 2 , i.e f (µ) ∝ 1, 1 . σ2 Thus, we can get the posterior sample of µ and σ 2 by Gibbs sampling and Metropolis-Hastings. For each fixed µ ˜ and σ ˜ 2 , we can get E[L(˜ µ, (µ, σ 2 ))] and E[L(˜ σ 2 , (µ, σ 2 ))] by MCMC. f (σ 2 ) ∝

ncsu-red

Bayesian intrinsic point estimation

10 / 17

c 2012 by Teng Zhang Fan Wu

Computation details

Simulation process Here, we choose four different pair parameters: µ = 0, 1, σ 2 = 1, 4. For each pair, our random sample will be n = 50. We calculate the three estimators through Gibbs sampling (N = 5000, burn in number is 1000). We repeat this process Nrep = 100 times, calculating the estimated Bias, variance and MSE. The result of simulation study is given as following,

ncsu-red

Bayesian intrinsic point estimation

11 / 17

c 2012 by Teng Zhang Fan Wu

Results

µ = 0, σ 2 = 1 Bias(µ) Var(µ) MSE(µ) Bias(σ 2 ) Var(σ 2 ) MSE(σ 2 )

Intrinsic 0.0021 0.0243 0.0004 0.0091 0.0474 0.0083

ncsu-red

Bayesian(Squared loss) 0.0018 0.0244 0.0003 0.0317 0.0494 0.1008

UMVUE 0.0016 0.0239 0.0003 -0.0108 0.0451 0.0116

Bayesian intrinsic point estimation

12 / 17

c 2012 by Teng Zhang Fan Wu

Results

µ = 1, σ 2 = 1 Bias(µ) Var(µ) MSE(µ) Bias(σ 2 ) Var(σ 2 ) MSE(σ 2 )

Intrinsic 0.0021 0.0243 0.0004 0.0091 0.0474 0.0083

ncsu-red

Bayesian(Squared loss) 0.0018 0.0244 0.0003 0.0317 0.0494 0.1008

UMVUE 0.0016 0.0239 0.0003 -0.0108 0.0451 0.0116

Bayesian intrinsic point estimation

13 / 17

c 2012 by Teng Zhang Fan Wu

Results

µ = 0, σ 2 = 4 Bias(µ) Var(µ) MSE(µ) Bias(σ 2 ) Var(σ 2 ) MSE(σ 2 )

Intrinsic 0.0045 0.0967 0.0020 0.0385 0.7557 0.1482

ncsu-red

Bayesian(Squared loss) 0.0045 0.0963 0.0021 0.1290 0.7899 1.6633

UMVUE 0.0033 0.0951 0.0011 -0.0431 0.7217 0.1857

Bayesian intrinsic point estimation

14 / 17

c 2012 by Teng Zhang Fan Wu

Results

µ = 1, σ 2 = 4 Bias(µ) Var(µ) MSE(µ) Bias(σ 2 ) Var(σ 2 ) MSE(σ 2 )

Intrinsic 0.0045 0.0967 0.0020 0.0385 0.7557 0.1482

ncsu-red

Bayesian(Squared loss) 0.0045 0.0966 0.0021 0.1290 0.7899 1.6633

UMVUE 0.0033 0.0955 0.0011 -0.0431 0.7217 0.1857

Bayesian intrinsic point estimation

15 / 17

c 2012 by Teng Zhang Fan Wu

Conclusions

Analysis of results The performance of Bias, variance, MSE does not depend on µ; The performance of three estimators of µ are similar; Bayesian estimator with squared loss is not good for estimating σ 2 ; Intrinsic estimator of σ 2 has smaller bias and MSE, but larger variance than UMVUE.

ncsu-red

Bayesian intrinsic point estimation

16 / 17

c 2012 by Teng Zhang Fan Wu

Conclusions

Pros and Cons Advantages: Invariant under transformation; good performance on Bias, variance and MSE. Disadvantages: Computation issues; low efficiency.

ncsu-red

Bayesian intrinsic point estimation

17 / 17

c 2012 by Teng Zhang Fan Wu

Reference

J.M. Bernardo. Objective bayesian point and region estimation in location-scale models. SORT-Statistics and Operations Research Transactions, 31(1):3–44, 2007. JM Bernardo and M. Juárez. Intrinsic estimation. Bayesian Statistics, 7:465–476, 2003. J.M. Bernardo and R. Rueda. Bayesian hypothesis testing: A reference approach.

ncsu-red

Bayesian intrinsic point estimation

17 / 17

c 2012 by Teng Zhang Fan Wu

Reference

International Statistical Review, 70(3):351–372, 2002. S. Kullback and R.A. Leibler. On information and sufficiency. The Annals of Mathematical Statistics, 22(1):79–86, 1951.

ncsu-red

Bayesian intrinsic point estimation

17 / 17

c 2012 by Teng Zhang Fan Wu