1

Parametric Estimation: Point Estimator and Interval Estimator The outcome of a coin toss follows a Bernoulli distribution f(x;2)=2x(1-2)1-x for x=0 or 1, where 2=P(X=1) is the probability of success (head). When the coin is known to be fair, 2=1/2 and we expect about 50% of n independent tosses will land on the head, and this fraction will get closer to 1/2 as the number of tosses increases. When the parameter of a distribution is unknown, we have two statistical inference problems: (a) Estimation - What is the value of 2? (b) Test of hypothesis - Is 2 equal to 1/2? Is it equal to 1/4? In the example above, the underlying population distribution is known to be a Bernoulli, and we only have to estimate its parameter 2. This is called the parametric estimation. In many application problems, the underlying distribution function is unknown a priori. For example, consider the distribution of the rents of apartments in a town, wage rates across workers, the rate of return from an investment, etc. In such cases, we choose a reasonable model such as a normal distribution, and estimate and test the unknown parameters of the chosen model. One may also wish to test whether the model is a proper one. Alternatively, one may wish to estimate the distribution function itself without assuming any particular distribution function. This last approach is called the non-parametric estimation. It is often the case that one wishes to estimate only the moments, such as mean and variance, without estimating the parameters of the distribution. Of course, the estimation of the moments can be equivalent to the estimation of the distribution parameters in some cases, e.g., the parameter 2 of Bernoulli distribution is also its mean. In other cases, the relationship between the distribution parameters and moments may not be that simple. Or, in the case of multiple variables, one may wish to estimate a model of linear function for the conditional mean without specific knowledge or estimation of the joint distribution function. To estimate the parameters of a distribution function f(x;2) we collect the sample from the population whose distribution function is f(x;2). For example, to estimate the parameter 2 of a Bernoulli pdf, toss a coin n times. The outcome of each toss is a Bernoulli random variable Xi, and they are mutually independent. The collection of n outcomes {X1,þ,Xn} is called a random sample of size n from an infinite population1 with a distribution function f(x;2). Definition. A random sample of size n from a population with a pdf f(x;2) is a set of n independent and identically distributed random variables, X1,þ,Xn with pdf f(x;2), such that the joint pdf of the random sample is given by

1

We can generate an infinite number of random variables by repeating the coin toss indefinitely. Each Xi can be considered as a sample from this infinite population. The major characteristics of an infinite population is that a sample withdrawn does not affect the probability distribution of the next sample. For a finite population, this is not true. For example, consider a box that contains k blue balls and N-k grey balls, but k is unknown We wish to estimate the number of blue balls. Draw a ball and record Xi=1 if blue and Xi=0 if grey. If sampling is done with replacement (sampling with replacement), then Xi’s are i.i.d. Bernoulli random variables with P(Xi=1)=k/N. In sampling without replacement, Xi’s are not independent. Sampling with replacement is sometimes called a sampling from an infinite population. We will study only the case of the infinite population.

2

where {x1,þ,xn} is a set of the realized values. ƒ Some authors call the realized observations {x1,þ,xn} a random sample. The random sample contains information about the underlying distribution function, from which the sample is taken. Realized observations of a random sample vary from one sample to another. Consider two random samples of size 10 from a Bernoulli population. The realized values of the first10 tosses of a coin can be different from the realized values of the second 10 tosses. This variation of the samples lead to the random variation of estimators we study below. Before we discuss various estimation methods, we will consider the properties of two sample statistics: sample mean and sample variance. Definition. Let Xi, i=1,þ,n, be a random sample of size n. The sample mean defined by

and the sample standard deviation is defined by

and sample variance S2 are

. ƒ

Theorem: Let Xi, i=1,þ,n, be a random sample of size n from a population f(x;:,F2), where : and F2