9/02/2014

Outline of my half-semester course:

1. Discrete time methods (Bellman Equation, Contraction Mapping Theorem, and Blackwell’s Suﬃcient Conditions, Numerical methods) • Applications to growth, search, consumption, asset pricing 2. Continuous time methods (Bellman Equation, Brownian Motion, Ito Process, and Ito’s Lemma) • Applications to search, consumption, price-setting, investment, industrial organization, asset-pricing

Outline of today’s lecture:

1. Introduction to dynamic programming

2. The Bellman Equation

3. Three ways to solve the Bellman Equation

4. Application: Search and stopping problem

1

Introduction to dynamic programming. • Course emphasizes methodological techniques and illustrates them through applications.

• We start with discrete-time dynamic optimization. • Is optimization a ridiculous model of human behavior? Why or why not? • Today we’ll start with an ∞-horizon stationary problem: The Sequence Problem (cf. Stokey and Lucas)

Notation: is the state vector at date ( +1) is the flow payoﬀ at date ( is ‘stationary’) is the exponential discount function is referred to as the exponential discount factor The discount rate is the rate of decline of the discount function, so ≡ − ln = −

h i

Note that exp(−) = and exp(−) =

Definition of Sequence Problem: Find () such that (0) =

sup

∞ X

{+1 }∞ =0 =0

( +1)

subject to +1 ∈ Γ() with 0 given. Remark 1.1 When I omit time subscripts, this implies that an equation holds for all relevant values of . In the statement above, +1 ∈ Γ() implies, +1 ∈ Γ() for all = 0 1 2

Example 1.1 Optimal growth with log utility and Cobb-Douglas technology: sup

∞ X

{}∞ =0 =0

ln()

subject to the constraints, ≥ 0 + +1 = and 0 given. Translate this problem into Sequence Problem notation by (1) eliminating redundant variables and (2) introducing constraint correspondence Γ Example 1.2 Optimal growth translated into Sequence Problem notation: (0) =

sup

∞ X

{+1}∞ =0 =0

ln( − +1)

such that +1 ∈ [0 ] ≡ Γ() and 0 given.

2

Bellman Equation

Compare Sequence Problem and Bellman Equation. Definition: Bellman Equation expresses the value function as a combination of a flow payoﬀ and a discounted continuation payoﬀ: () =

sup +1∈Γ()

{ ( +1) + (+1)}

∀

• Flow payoﬀ is ( +1) • Current value function is () Continuation value function is (+1) • Equation holds for all (feasible) values of

• We call (·) the solution to the Bellman Equation. • Note that any old function won’t solve the Bellman Equation. • We haven’t yet demonstrated that there exists even one function (·) that will satisfy the Bellman equation.

• We will show that the (unique) value function defined by the Sequence Problem is also the unique solution to the Bellman Equation.

A solution to the Sequence Problem is also a solution to the Bellman Equation. (0) = =

∞ X

sup

+1∈Γ() =0 ⎧ ⎨

sup

+1∈Γ()

= = =

sup

(0 1) +

⎩ ⎧ ⎨

1∈Γ(0)

sup

1∈Γ(0)

∞ X

=1 ∞ X

(0 1) +

⎩

⎫ ⎬

( +1)

(0 1) +

⎩ +1∈Γ() ⎧ ⎨

sup

( +1)

⎭

⎫ ⎬

−1 ( +1)

=1

sup

∞ X

+1 ∈Γ() =0

{ (0 1) + (1)}

⎭

⎫ ⎬

(+1 +2)

⎭

A solution to the Bellman Equation is also a solution to the Sequence Problem. (0) = =

sup 1∈Γ(0)

{ (0 1) + (1)}

sup

+1∈Γ()

..

=

sup +1∈Γ()

=

sup

{ (0 1) + [ (1 2) + (2)]} n

o −1 (0 1) + · · · + (−1 ) + ()

∞ X

+1∈Γ() =0

( +1)

Suﬃcient condition: lim→∞ () = 0 ∀ feasible sequences (Stokey and Lucas Thm. 4.3). In summary, a solution to the Bellman Equation will also be a solution to the Sequence Problem and vice versa.

Example 2.1 Optimal growth in Sequence Problem notation: (0) =

sup

∞ X

{+1}∞ =0 =0

ln( − +1)

such that +1 ∈ [0 ] ≡ Γ() and 0 given. Optimal growth in Bellman Equation notation: () =

sup +1∈Γ()

{ln( − +1) + (+1)}

∀

3

Solving the Bellman Equation • Three methods 1. guess a solution (that’s no typo) 2. iterate functional operator analytically (what’s a functional operator?) 3. iterate functional operator numerically

• Method 1 today. • Guess a function (), and then check to see that this function satisfies the Bellman Equation at all possible values of

• For our growth example, guess that the solution of the growth problem takes the form: () = + ln() where and are constants for which we need to find solutions.

• Here value function inherits functional form of utility function (ln). • To solve for constants rewrite Bellman Equation: () =

sup +1∈Γ()

+ ln() =

sup +1∈Γ()

{ln( − +1) + (+1)}

∀

{ln( − +1) + [ + ln(+1)]}

∀

First order condition (FOC) on the right-hand-side of the Bellman Equation: ( +1) + 0(+1) = 0 +1 Envelope Theorem: ( +1) 0 () =

Heuristic Proof of Envelope Theorem: ( +1) ( +1) +1 + + 0(+1) +1 +1 " # ( +1) +1 ( +1) + 0(+1) = + +1 ( +1) =

0() =

Problem Set 1 asks you to use the FOC and the Envelope Theorem to solve for and . You will also confirm that () = + ln() is a solution to the Bellman Equation.

4

Search and optimal stopping

Example 4.1 An agent draws an oﬀer, from a uniform distribution with support in the unit interval. The agent can either accept the oﬀer and realize net present value (ending the game), or the agent can reject the oﬀer and draw again a period later. All draws are independent. Rejections are costly because the agent discounts the future exponentially with discount factor . This game continues until the agent receives an oﬀer that she is willing to accept.

• The Bellman equation for this problem is (relatively) easy to write: () = max{ [(+1)]}

(1)

Our problem is to find the value function (·) that solves equation (1). We’ll also want to find the associated policy rule. Definition: A policy is a function that maps to the action space. Definition: An optimal policy achieves payoﬀ () for all feasible .

Proposition: In the search and optimal stopping problem, the threshold policy with cutoﬀ ∗ is a best response to any continuation value function, b if and only if (iﬀ) ∗ = [b(+1)]

Proof: Optimization generates the following policy: ACCEPT iﬀ ∗ = [b(+1)] REJECT iﬀ ∗ = [b(+1)]

If = ∗ = [b(+1)] then ACCEPT and REJECT generate the same payoﬀ. ¥

• Find threshold ∗ so that the associated value function, () =

(

if ≥ ∗ ∗ if ≤ ∗

)

(2)

satisfies the Bellman Equation. • In other words, find the value of ∗ so that () (defined in equation 2) solves the Bellman Equation (equation 1).

If = ∗ you should be indiﬀerent between stopping and continuing. (∗) = ∗ = (+1) =

Z =∗

∗ () +

=0 1 1 ∗ 2 = ( ) +

2

So the final result is

Z =1

=∗

()

2

i h ∗ 2 ∗ ( ) + 1 =

2

which has solution

µ

∗ = −1 1 −

q

¶

1 − 2

Always think about comparative statics and sensibility of the answer.

Optimal threshold in stopping problem 1 0.9 converges to 1 as discount rate goes to 0

0.8

optimal threshold

0.7 0.6 0.5 0.4 0.3 converges to 0 as discount rate goes to ∞

0.2 0.1 0

0.1

0.2

0.3

0.4 0.5 0.6 0.7 discount rate = -ln(delta)

0.8

0.9

1

Outline of today’s lecture:

1. Introduction to dynamic programming

2. The Bellman Equation

3. Three ways to solve the Bellman Equation

4. Application: Search and stopping problem