Introduction Strategies analysis Example: Futures trading Conclusion
A New Approach to Estimating the Bellman Function Jan Zeman Institute of Information Theory and Automation Prague, Czech Republic
September 22, 2009
Jan Zeman
A New Approach to Estimating the Bellman Function
Introduction Strategies analysis Example: Futures trading Conclusion
Dynamic programming Bellman function
Outline 1
Introduction Dynamic programming Bellman function
2
Strategies analysis Off-line strategies Similarity indexes Estimating the Bellman function
3
Example: Futures trading Task definition Similarity indexes Experiment
4
Conclusion Jan Zeman
A New Approach to Estimating the Bellman Function
Introduction Strategies analysis Example: Futures trading Conclusion
Dynamic programming Bellman function
Dynamic programming System - part of world interesting for decision maker Decision maker - human or machine with aims according to system Decisions xt is designed by decision maker to reach the aims Output yt is information about system available to decision maker Gain function - degree of reaching the aims G : (x1 , . . . , xT , y1 , . . . , yT ) → R+ 0 The main aim - search the sequence {x1 , . . . , xT } to maximize: max
{x1 ,...,xT } Jan Zeman
G=
max
{x1 ,...,xT }
T X
gk .
k=1
A New Approach to Estimating the Bellman Function
Introduction Strategies analysis Example: Futures trading Conclusion
Dynamic programming Bellman function
Dynamic programming - Bellman function Off-line optimization at the time t: Vt =
T X
max
{xt ,...,xT }
gk
k=t
Bellman function (recursive shape): Vt = max(gt + Vt+1 ) xt
Optimal decision: xt = arg max(gt + Vt+1 ) xt
Jan Zeman
A New Approach to Estimating the Bellman Function
Introduction Strategies analysis Example: Futures trading Conclusion
Dynamic programming Bellman function
Dynamic programming - Bellman function On-line optimization at the time t: ! T X Vt = max E gk x1 , . . . , xt−1 , y1 , . . . , yt {xt ,...,xT } k=t
Bellman function (recursive shape): Vt = max xt
E (gt + Vt+1
|x1 , . . . , xt−1 , y1 , . . . , yt )
Optimal decision: xt = arg max xt
E (gt + Vt+1
Jan Zeman
|x1 , . . . , xt−1 , y1 , . . . , yt )
A New Approach to Estimating the Bellman Function
Introduction Strategies analysis Example: Futures trading Conclusion
Off-line strategies Similarity indexes Estimating the Bellman function
Off-line strategy analysis Assumption The decision xt does not influence the system. The optimal strategy obtained at whole dataset: X T = (x1T , x2T , . . . , xTT ) Optimal strategies for shorter horizon: X 1 = (x11 ) X 2 = (x12 , x22 ) X 3 = (x13 , x23 , x33 ) .. . XT
= (x1T , x2T , x3T , x4T , . . . , xTT ) Jan Zeman
A New Approach to Estimating the Bellman Function
Introduction Strategies analysis Example: Futures trading Conclusion
Off-line strategies Similarity indexes Estimating the Bellman function
Similarity indexes Similarity index (number of similar decisions): St =
t X
δ(xit , xiT )
i=1
Strict similarity index (length of non-broken similarity): st = max{i; (∀j ∈ N )(j ≤ i ⇒ xjt = xjT )} i
st ≤ St ≤ t Example: Xt
= XT =
1 {1 {1
2 -1 -1
3 0 0
4 1 1
5 0 1
st = 4, Jan Zeman
6 1 1
7 0 0
8 ...t 0 ...0 1 ...1
} ...}
St = 6 A New Approach to Estimating the Bellman Function
Introduction Strategies analysis Example: Futures trading Conclusion
Off-line strategies Similarity indexes Estimating the Bellman function
Estimating the Bellman function Bellman function (recursive shape): Vt = max xt
E (gt + Vt+1 |x1 , . . . , xt−1 , y1 , . . . , yt )
If st ≈ t and we insert the X t , we obtain system of functional equations: t Vk = max E gk + Vk+1 |x1t , . . . , xk−1 , y1 , . . . , yk for k ∈ {1, . . . st }, xk
which can be transformed to system algebraic equations for parametrized shape of Bellman function. Solution is inserted into on-line Bellman equation and the maximization can be calculated. Jan Zeman
A New Approach to Estimating the Bellman Function
Introduction Strategies analysis Example: Futures trading Conclusion
Task definition Similarity indexes Experiment
Futures trading: task definition Futures futures contract, obligation to buy a normalized amount of a commodity Speculator chooses the decision about the future Long - believe in price increase Short - believe in price decrease Flat - do not believe - out of market Gain function: G=
T X k=1
y1 , . . . , yT where x1 , . . . , xT C
(y − yk−1 )xt−1 − C |xk−1 − xk |, {z } | k gk
price of contract count of held contracts transaction cost per contract Jan Zeman
A New Approach to Estimating the Bellman Function
Introduction Strategies analysis Example: Futures trading Conclusion
Task definition Similarity indexes Experiment
Calculated variables Five reference price sequences: Cocoa (CC) Light crude oil (CL) U.S. Treasure note (FV2) Japanese Yen (JY) Wheat (W) Calculated variables: c1 = maxt (t − st ) c2 = maxt (t − St ) tch,1 and tch,2 - last change of value c1 and c2
Jan Zeman
A New Approach to Estimating the Bellman Function
Introduction Strategies analysis Example: Futures trading Conclusion
Task definition Similarity indexes Experiment
Table of results
Market CC CL FV2 JY W
c1 6 444 8 4 7
c2 6 6 8 4 7
tch;1 342 847 383 50 2452
tch;2 342 2205 383 50 2452
T 3822 3863 3766 3871 3822
Table: Dominating constants c1 and c2
Jan Zeman
A New Approach to Estimating the Bellman Function
Introduction Strategies analysis Example: Futures trading Conclusion
3000
2640
st St
2500
Task definition Similarity indexes Experiment
2620
2000
2600
1500
2580
st St t
2560
1000
2540
500
2520
0 0
500 1000 1500 2000 2500
Jan Zeman
2500 2500 2520 2540 2560 2580 2600
A New Approach to Estimating the Bellman Function
Introduction Strategies analysis Example: Futures trading Conclusion
3000
2300
st St
2500
Task definition Similarity indexes Experiment
2280 2260
2000
st St t
2240
1500
2220
1000
2200
500
2180 2160
0 0
500 1000 1500 2000 2500
Jan Zeman
2160 2180 2200 2220 2240
A New Approach to Estimating the Bellman Function
Introduction Strategies analysis Example: Futures trading Conclusion
Task definition Similarity indexes Experiment
Experiment setup
Experiment: Parametrized shape of Bellman function Weighted least squares method Prediction using autoregressive model ⇒ iteration spread in time (IST) Reference results: Model predictive method (MPC) Predictive model and task setup were same as above
Jan Zeman
A New Approach to Estimating the Bellman Function
Introduction Strategies analysis Example: Futures trading Conclusion
Task definition Similarity indexes Experiment
Results
Market CC CL FV2 JY W
MPC -6 450 -12 350 -5 701 -26 568 -9 792
-1 3 10 -35 -1
IST 490 390 727 247 923
Table: Results of experiment
Jan Zeman
A New Approach to Estimating the Bellman Function
Introduction Strategies analysis Example: Futures trading Conclusion
Conclusion
IST is better in 4 of 5 datasets. The approach needs further testing. The similarity indexes are calculated non-causally. The causal criterion of usage the approach should be find.
Jan Zeman
A New Approach to Estimating the Bellman Function