A New Approach to Estimating the Bellman Function

Introduction Strategies analysis Example: Futures trading Conclusion A New Approach to Estimating the Bellman Function Jan Zeman Institute of Informa...
Author: Eileen Ball
4 downloads 0 Views 212KB Size
Introduction Strategies analysis Example: Futures trading Conclusion

A New Approach to Estimating the Bellman Function Jan Zeman Institute of Information Theory and Automation Prague, Czech Republic

September 22, 2009

Jan Zeman

A New Approach to Estimating the Bellman Function

Introduction Strategies analysis Example: Futures trading Conclusion

Dynamic programming Bellman function

Outline 1

Introduction Dynamic programming Bellman function

2

Strategies analysis Off-line strategies Similarity indexes Estimating the Bellman function

3

Example: Futures trading Task definition Similarity indexes Experiment

4

Conclusion Jan Zeman

A New Approach to Estimating the Bellman Function

Introduction Strategies analysis Example: Futures trading Conclusion

Dynamic programming Bellman function

Dynamic programming System - part of world interesting for decision maker Decision maker - human or machine with aims according to system Decisions xt is designed by decision maker to reach the aims Output yt is information about system available to decision maker Gain function - degree of reaching the aims G : (x1 , . . . , xT , y1 , . . . , yT ) → R+ 0 The main aim - search the sequence {x1 , . . . , xT } to maximize: max

{x1 ,...,xT } Jan Zeman

G=

max

{x1 ,...,xT }

T X

gk .

k=1

A New Approach to Estimating the Bellman Function

Introduction Strategies analysis Example: Futures trading Conclusion

Dynamic programming Bellman function

Dynamic programming - Bellman function Off-line optimization at the time t: Vt =

T X

max

{xt ,...,xT }

gk

k=t

Bellman function (recursive shape): Vt = max(gt + Vt+1 ) xt

Optimal decision: xt = arg max(gt + Vt+1 ) xt

Jan Zeman

A New Approach to Estimating the Bellman Function

Introduction Strategies analysis Example: Futures trading Conclusion

Dynamic programming Bellman function

Dynamic programming - Bellman function On-line optimization at the time t: ! T X Vt = max E gk x1 , . . . , xt−1 , y1 , . . . , yt {xt ,...,xT } k=t

Bellman function (recursive shape): Vt = max xt

E (gt + Vt+1

|x1 , . . . , xt−1 , y1 , . . . , yt )

Optimal decision: xt = arg max xt

E (gt + Vt+1

Jan Zeman

|x1 , . . . , xt−1 , y1 , . . . , yt )

A New Approach to Estimating the Bellman Function

Introduction Strategies analysis Example: Futures trading Conclusion

Off-line strategies Similarity indexes Estimating the Bellman function

Off-line strategy analysis Assumption The decision xt does not influence the system. The optimal strategy obtained at whole dataset: X T = (x1T , x2T , . . . , xTT ) Optimal strategies for shorter horizon: X 1 = (x11 ) X 2 = (x12 , x22 ) X 3 = (x13 , x23 , x33 ) .. . XT

= (x1T , x2T , x3T , x4T , . . . , xTT ) Jan Zeman

A New Approach to Estimating the Bellman Function

Introduction Strategies analysis Example: Futures trading Conclusion

Off-line strategies Similarity indexes Estimating the Bellman function

Similarity indexes Similarity index (number of similar decisions): St =

t X

δ(xit , xiT )

i=1

Strict similarity index (length of non-broken similarity): st = max{i; (∀j ∈ N )(j ≤ i ⇒ xjt = xjT )} i

st ≤ St ≤ t Example: Xt

= XT =

1 {1 {1

2 -1 -1

3 0 0

4 1 1

5 0 1

st = 4, Jan Zeman

6 1 1

7 0 0

8 ...t 0 ...0 1 ...1

} ...}

St = 6 A New Approach to Estimating the Bellman Function

Introduction Strategies analysis Example: Futures trading Conclusion

Off-line strategies Similarity indexes Estimating the Bellman function

Estimating the Bellman function Bellman function (recursive shape): Vt = max xt

E (gt + Vt+1 |x1 , . . . , xt−1 , y1 , . . . , yt )

If st ≈ t and we insert the X t , we obtain system of functional equations:  t Vk = max E gk + Vk+1 |x1t , . . . , xk−1 , y1 , . . . , yk for k ∈ {1, . . . st }, xk

which can be transformed to system algebraic equations for parametrized shape of Bellman function. Solution is inserted into on-line Bellman equation and the maximization can be calculated. Jan Zeman

A New Approach to Estimating the Bellman Function

Introduction Strategies analysis Example: Futures trading Conclusion

Task definition Similarity indexes Experiment

Futures trading: task definition Futures futures contract, obligation to buy a normalized amount of a commodity Speculator chooses the decision about the future Long - believe in price increase Short - believe in price decrease Flat - do not believe - out of market Gain function: G=

T X k=1

y1 , . . . , yT where x1 , . . . , xT C

(y − yk−1 )xt−1 − C |xk−1 − xk |, {z } | k gk

price of contract count of held contracts transaction cost per contract Jan Zeman

A New Approach to Estimating the Bellman Function

Introduction Strategies analysis Example: Futures trading Conclusion

Task definition Similarity indexes Experiment

Calculated variables Five reference price sequences: Cocoa (CC) Light crude oil (CL) U.S. Treasure note (FV2) Japanese Yen (JY) Wheat (W) Calculated variables: c1 = maxt (t − st ) c2 = maxt (t − St ) tch,1 and tch,2 - last change of value c1 and c2

Jan Zeman

A New Approach to Estimating the Bellman Function

Introduction Strategies analysis Example: Futures trading Conclusion

Task definition Similarity indexes Experiment

Table of results

Market CC CL FV2 JY W

c1 6 444 8 4 7

c2 6 6 8 4 7

tch;1 342 847 383 50 2452

tch;2 342 2205 383 50 2452

T 3822 3863 3766 3871 3822

Table: Dominating constants c1 and c2

Jan Zeman

A New Approach to Estimating the Bellman Function

Introduction Strategies analysis Example: Futures trading Conclusion

3000

2640

st St

2500

Task definition Similarity indexes Experiment

2620

2000

2600

1500

2580

st St t

2560

1000

2540

500

2520

0 0

500 1000 1500 2000 2500

Jan Zeman

2500 2500 2520 2540 2560 2580 2600

A New Approach to Estimating the Bellman Function

Introduction Strategies analysis Example: Futures trading Conclusion

3000

2300

st St

2500

Task definition Similarity indexes Experiment

2280 2260

2000

st St t

2240

1500

2220

1000

2200

500

2180 2160

0 0

500 1000 1500 2000 2500

Jan Zeman

2160 2180 2200 2220 2240

A New Approach to Estimating the Bellman Function

Introduction Strategies analysis Example: Futures trading Conclusion

Task definition Similarity indexes Experiment

Experiment setup

Experiment: Parametrized shape of Bellman function Weighted least squares method Prediction using autoregressive model ⇒ iteration spread in time (IST) Reference results: Model predictive method (MPC) Predictive model and task setup were same as above

Jan Zeman

A New Approach to Estimating the Bellman Function

Introduction Strategies analysis Example: Futures trading Conclusion

Task definition Similarity indexes Experiment

Results

Market CC CL FV2 JY W

MPC -6 450 -12 350 -5 701 -26 568 -9 792

-1 3 10 -35 -1

IST 490 390 727 247 923

Table: Results of experiment

Jan Zeman

A New Approach to Estimating the Bellman Function

Introduction Strategies analysis Example: Futures trading Conclusion

Conclusion

IST is better in 4 of 5 datasets. The approach needs further testing. The similarity indexes are calculated non-causally. The causal criterion of usage the approach should be find.

Jan Zeman

A New Approach to Estimating the Bellman Function

Suggest Documents