c 2013 IEEE THE FINAL VERSION OF THIS MANUSCRIPT APPEARED IN THE PROCEEDINGS OF THE IEEE ICC 2013, 9-13 JUNE 2013, BUDAPEST, HUNGARY. 1

Energy Storage Optimization Strategies for Smart Grids Claudio G. Codemo, Tomaso Erseghe, Andrea Zanella

Abstract—The efficient management of the supply and demand in electricity networks is becoming a pivotal issue with important fallbacks both in the technological and financial domains. An interesting topic in this domain is the use of large batteries at the end users premises to reduce the average cost of energy supply, by storing energy when its cost is low and releasing it when the cost is high. In this paper, we wish to gain insights on the impact of some system model parameters, such as battery capacity, charge/discharge rate, power request process, and cost functions, on the cost saving that can be achieved by some selected energy storage algorithms. The study shows that the battery capacity has a direct and rather linear impact on cost reduction, while the effect of charge/discharge rates is less straightforward to predict. Furthermore, we show that, with piecewise, convex and nondecreasing cost functions, the optimal energy storage strategy has a threshold structure, where the number of thresholds depend on the shape of the cost function and the constraints of the battery. Index Terms—SmartGrid, Energy Storage, Optimization, Dynamic-Programming, Single-Threshold Algorithm

I. I NTRODUCTION The Smart Grid is an attempt at modernizing the current existing electric grid by incorporating modern digital communication technologies to ensure efficient use of the electric energy and reduce the cost for energy supply. Smart Grids could exploit the fact that the cost of energy production and delivery undergoes significant fluctuations during each day due to variations in demand and generator capacity. These fluctuations are traditionally hidden to the end users, who pay a fixed retail energy price. However, dynamic pricing creates an opportunity for users, such as households or data centers, to reduce energy costs by exploiting the price fluctuations [1]. Unfortunately, in practice, users’ power demands are not much reactive to changes in the energy prices [2]–[5]. A possible solution is to equip single users, or small groups organized in a micro-grid, with an energy storage device, e.g., a large battery, that can store energy when the price is low and release it when the price is high. This allows users to benefit from the energy price variations without having to adjust their consumption. There exists a significant amount of work on energy storage strategies in various contexts: among them, our starting points are [6] and [7]. In [6], the authors address the optimal energy storage control problem that is faced by a grid operator, whose controller has access to one energy storage device of finite This work is partially funded by the PRIN 2009 “Alter-Net” project. c 2013 IEEE. Personal use of this material is permitted. Permission from

IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.

storage capacity. The simple Single-Threshold (ST) policy proposed by the authors is proved to be be asymptotically optimal as the battery storage grows to infinity. However, it does not take into account realistic charge and discharge rates of the battery, nor a realistic power demand process, which is arbitrarily modeled as the state of a M/M/∞ queueing system. The problem of optimal control of the battery charge/discharge process to minimize the long-term average cost is investigated in [7] as a Markov Decision Process (MDP): under rather general assumptions, the optimal policy is shown to have a two-thresholds structure. Thresholds are numerically computed in some specific scenarios, using real-world power request traces from existing energy markets. The results of [7], however, have been obtained under the assumption that the energy cost function is always linear with respect to (wrt) power demand requests, a model that does not capture the more-than-linear increase of the cost to produce and delivery energy as the request grows. In this paper, we wish to shed some light on the effect that different model parameters may have on the performance of energy storage strategies. To this aim, we consider realistic power request traces, obtained from the high-detailed models proposed in [8]. For the energy storage device, i.e., the battery, we adopt a very simple and abstract model that, however, keeps into account the most important physical limitations of these devices, such as finite storage capacity, limited charge/discharge rates, power losses during charging/discharging operations. Charge-retention and battery cost depreciation in time have also been considered in our study but are not included in this paper because their effect has turned out to be either negligible or not significant in our scenario. Concerning the energy cost function, we consider an always increasing, convex, piecewise linear cost function similar to that proposed in [6] that reflects the more than linear increase in the cost of producing and delivering each additional power unit. Nonetheless, to get insights on the relation between the shape of the cost function on the optimal energy storage strategy, we consider three cost functions that, while sharing the same piecewise convex structure, differ for the number and/or slope of linear segments. The study considers four different energy storage strategies, named A-posteriori Optimal Strategy (AOS), Dynamic Programming with Markovian requests (DPM), Dynamic Programming with Independent request (DPI), and SingleThreshold (ST) algorithm, respectively. We selected these strategies because they require progressively less prior information on the statistic of the user’s power requests but, on the other hand, offer progressively diminishing cost saving with respect to a reference system with no battery. Comparing

II. S YSTEM MODEL We consider the system schematically represented in Fig. 1. User represents the power-demanding entity, Grid is the main power supplier, while Battery is a local energy storage device that the user may exploit to adjust the overall power request to the grid, in order to minimize the average cost payed for the power supply. The battery charge/discharge policy is decided by the Storage Control Unit (SCU) on the basis of the optimization algorithms that will be presented in Section III. Emax

Battery ηc E(t) δ(t)

0

ηd

Storage Control Unit (SCU)

X(t)

Grid

L(t) User

Fig. 1.

L

0.1



the performance of the four algorithms when varying the constraints on the battery model and the cost functions we can gain some insights on the structure of the optimal charge/discharge policy in different conditions. The remainder of the paper is organized as follows. In Sec. II we introduce the system model, specifying the battery, power request, and cost models, and formalizing the optimization problem. Sec. III describes the four optimization strategies considered in the analysis, whose results are presented and discussed in Sec. IV, which concludes the paper.

PDF

c 2013 IEEE THE FINAL VERSION OF THIS MANUSCRIPT APPEARED IN THE PROCEEDINGS OF THE IEEE ICC 2013, 9-13 JUNE 2013, BUDAPEST, HUNGARY. 2

0.05

0 0

Fig. 2.

10

20

30

40

50

[kW]

60

Empirical PDF of the power requests L(t).

We used the aforementioned model provided by [8] to generate different traces of power requests from a group of 20 users over the 31 days of January. Data have been aggregated with the granularity of one hour, which is a reasonable time interval to capture the dynamic of the aggregate synthetic power request process obtained from model [8]. The empirical probability density function (PDF) of the resulting power request process L(t) is plotted in Fig. 2, together with its mean that, under ergodic assumption, is equal to L = E[L(t)]. B. Battery model The battery is modeled as a sort of power buffer, which can store and/or release units of power at each slot, upon request. We denote by E(t) the number of power units stored in the battery at time t, i.e., the battery energy level normalized to the time slot. Denoting by Emax the maximum storage capacity of the battery, it holds

Reference system architecture

0 ≤ E(t) ≤ Emax ,

∀t .

(1)

Below we describe in greater details the power request model, the battery model and the cost function.

The one-slot variation of the battery charge is defined as

A. Power request model We assume that time is slotted, i.e., t ∈ {0, 1, . . .}. At every slot, the user generates an overall power request, L(t), that must be sorted out in that very slot. If L(t) > 0 the user is absorbing power from the system, thus acting as a load. In this case, the requested power L(t) must be provided by the grid and/or the battery. On the contrary case, i.e., when L(t) < 0, the user is producing an excess of power (e.g., by means of renewable energy sources) that is either sold to the grid, or used to charge the battery, or a combination of the two. In general, the power request process may be influenced by some environmental parameters, as the period of the year, the weather conditions, and so on. In this paper, however, we neglect this aspect, under the assumption that the environmental dynamic is much slower than the system dynamic. Instead, we generate the power request process by using the sophisticated and realistic model proposed in [8]. The model makes it possible to generate a synthetic power demand signal, with one-minute time granularity, by simulating the typical patterns of power requests by common domestic appliances. The model has been validated by way of a comprehensive comparison of the statistical characteristics of synthetic data and empirical data collected on 22 domestic dwellings in UK.

where δmin ≤ 0 and δmax ≥ 0 are the maximum rates of discharge and charge, respectively. Note that δ is positive when the battery is charged, absorbing power from the grid, whereas is negative when the battery delivers power to the system. However, in real systems a certain fraction of power is dissipated during each charge/discharge operation, so that the net power requested to the grid at slot t is given by

δ(t) = E(t+1)−E(t) ,

subject to δmin ≤ δ(t) ≤ δmax (2)

X(t) = L(t) + f (δ(t)) where

( f (δ) =

δ/ηc , δ ≥ 0 (charge) δ ηd , δ < 0 (discharge)

(3)

(4)

where 0 < ηc < 1 is the efficiency of charge, reflecting the fact that only a portion of the energy given to the battery in a certain interval of time causes an actual increase of its level of energy; 0 < ηd < 1, similarly, represents an analogous effect during the discharge process. C. Cost model The instantaneous operator cost associated with power consumption, X(t), is denoted as C (X(t)), where C(·) is a nondecreasing and convex function. Convexity reflects the fact

c 2013 IEEE THE FINAL VERSION OF THIS MANUSCRIPT APPEARED IN THE PROCEEDINGS OF THE IEEE ICC 2013, 9-13 JUNE 2013, BUDAPEST, HUNGARY. 3

A. A-posteriori Optimal Strategy (AOS) C2 (`)





100

C3 (`)



cost units

150

50

C1 (`) 0 0

Fig. 3.

10

20

30

40

50

` [kW]

60

The three piecewise-linear cost functions considered.

that the marginal reward of the power supplier (grid) must increase as the overall power demand increases, to compensate for the progressively higher costs of producing and delivering each additional unit of power to the user. The cost function can then be approximated as a piecewise linear function, as represented in the three functions, C1 , C2 , and C3 , of Fig. 3 that are considered as reference in the rest of the paper. These functions have been selected in order to provide the same mean cost in a system without energy storage, with a power request process distribution as in Fig. 2. D. Problem statement Assuming a temporal horizon of T time slots, the objective is to find the battery charge/discharge strategy, δ(t), that minimizes the overall cost paid by the users within the grid to satisfy their power request process L(t). In formula, we thus have T X min C(X(t)) δ(t) (5) t=1 subject to (1)-(4), L(t), and E(0) = E0 where E0 is the initial energy level, while δ(t) will depend, in general, on the battery level E(t) and the current power request L(t). In practical settings, L(t) is not known a priori, and the problem (5) has to be solved “on-line”, as the system evolves. A possible approach in this case is to periodically estimate the statistics of L(t), supposing that the stochastic process can be well-approximated as stationary in the meanwhile, and reformulate (5) as min δ(t)

T X

E [C(X(t))]

t=1

(6)

subject to (1)-(4), PL , and E(0) = E0 where PL is a statistical description of the process L(t). III. O PTIMIZATION STRATEGIES In this section we describe some possible solutions of the minimization problem (5) that are all optimal under different assumptions on the statistical distribution of L(t). Furthermore, we briefly recall the Single-Threshold (ST) strategy, proposed in [6], which has been proved to be asymptotically optimal for any distribution of L(t) as the battery capacity and the charge/discharge rates grow to infinity.

AOS is an ideal strategy that solves (5) under the assumption that the whole process {L(t), t = 1, . . . , T } is known beforehand. This assumption is obviously unrealistic in practical scenarios, but it provides a performance upper bound on the cost saving that can be reached by any practical optimization algorithm. Knowing L(t) a priori, the optimal solution of (5) can be solved in an efficient manner by applying a Viterbi-like algorithm, with a computational complexity that grows linearly with T . This requires to quantize the battery level in a finite number NE of states, thus introducing a granularity of QE = Emax /NE in the energy levels. The state of the Viterbi algorithm at any given step k will then be expressed by the minimum cost to reach each of the N possible quantized battery levels, with power requests {L(t), t = 1, . . . , k}. For each battery level, En , then the algorithm evaluates the additional cost to reach, in the next step, any of the admissible battery levels Em = En + δ, with δmin ≤ δ ≤ δmax and 0 ≤ Em ≤ Emax . After T steps, the algorithm returns the sequence of charge/discharge actions δ(t) that yields the least final cost. B. Dynamic Programming with Markovian requests (DPM) The DPM approach assumes L(t) to be Markovian, so that the system state at any given time t is completely determined by the pair (e, `) of the battery level and power request at time t . The optimal solution to (6) can then be expressed as a function π(e, `) that assigns to each pair (e, `) an optimal charge/discharge action δ ? in the admissible range A(e) = {δ : δ ∈ [δmin , δmax ], e + δ ∈ [0, Emax ]}. Formally, let Gt (e, `) be the optimal average cost the user is expected to pay in the time interval t to T , by applying the optimal storage strategy, given that the battery level and the power request at time t are e and `, respectively. We can then write the following Belmann’s equation   Z Gt (e, `) = min C(`+f (δ)) + Gt+1 (e + δ, h)PL (h|`)dh δ∈A(e)

(7) where PL (h|`) is the one-step transition probability from L(t) = ` to L(t + 1) = h. This recursion can be solved starting from the time horizon T and going backward to zero. If the time horizon is sufficiently large, the value of δ that attains the minimum in (7) for any given state (e, `) converges to a time-independent function π(e, `), which is the optimal strategy. In this paper, we have estimated PL from the synthetic power request sequences generated according to the model [8], which is not perfectly Markovian, so that DPM optimality is not longer guaranteed. Nonetheless, the accuracy of the Markovian approximation is generally considered to be sufficient to achieve quasi-optimal performance. Otherwise, it is possible to increase the accuracy of the model by including other parameters, such as the time of the day, in the system state, at the cost of a larger state space and, in turn, higher computational complexity.

c 2013 IEEE THE FINAL VERSION OF THIS MANUSCRIPT APPEARED IN THE PROCEEDINGS OF THE IEEE ICC 2013, 9-13 JUNE 2013, BUDAPEST, HUNGARY. 4

C. Dynamic Programming with Independent request (DPI) The DPI approach further simplifies the DPM by assuming that L(t) is an independent identically distributed (iid) stochastic process. In this case, the transition probability matrix is such that PL (h, `) has all identical columns, each corresponding to the stationary probability distribution of L(t). The optimal strategy is still obtained by resolving (7) that, however, simplifies into  ¯ t+1 (e + δ) (8) Gt (e, `) = min C(`+f (δ)) + G

(a)

γ

20

D. Single-Threshold (ST) algorithm The ST algorithm [6] is extremely simple and only requires the mean L of L(t). The basic idea is that, as long as the energy storage device has finite capacity, the mean power drawn from the R T grid will always equal L on the long run, i.e., limT →∞ 0 X(t)dt = L. Assuming the cost function is convex in X(t), it is easy to realize that the minimum average cost is attained when X(t) = L for all t. Therefore, the strategy of the ST algorithm is to charge/discharge the battery in order to perfectly compensate for the variations of L(t) around its mean value. Considering the power losses incurred any time the battery is operated, the ST strategy can then be expressed as
δ(t) = f −1 max{L − L(t), −E(t)} (9) where f −1 (·) is the inverse of the function f (δ) given by (4). This strategy has been proved to be optimal when both the battery capacity and the charge/discharge rates tend to infinity. Considering a more realistic battery model, however, (9) cannot be always satisfied, since δ must fall in the feasible set A(E(t)). Therefore, the actual power demand to the grid will not be always equal to L, thus yielding larger costs. It is therefore interesting to assess the performance loss incurred by this simple algorithm when applied to a system with realistic constraints. IV. S IMULATION RESULTS In this section we compare the performance achieved by the four algorithms AOS, DPM, DPI, and ST in terms of cost improvement factor with respect to the reference case of a system with no battery, formally defined as Cal (10) Cno where Cno is the cost in the absence of battery, and Cal the cost for algorithm “al”. Note that, the three cost functions of Fig. 3 have been designed in order to attain the same cost Cno , so that the performance comparison is fair also under different cost scenarios. In the following we assume T = 5000 samples, symmetrical discharge rates, δmax = −δmin , a starting energy level E(0) = 1 2 Emax , and symmetric charge/discharge efficiency coefficients η = ηc = ηd = 0.95. Provided that the time horizon is not too small [9], the results are independent of the value of T and E(0). Note also that performance scales linearly with η. The γal = 1 −

(c)

22

22

γ

AOS DPM DPI ST

γ

20

20

[%]

[%]

18

18

16

16

16

14

14

12

12

12

10

10

8

8

6

6

6

4

4

4

2

2

2

[%] 18

10

δ∈A(e)

¯ t (e) is the expectation of Gt (e, `) wrt `. where G

(b)

22

8

0 0

Emax = 6 kWh

0.5

δmax /Emax

1

0 0

14

Emax = 12 kWh

0.5

Emax = 24 kWh

1

δmax /Emax

0 0

0.5

1

δmax /Emax

Fig. 4. Cost improvement (in percentage) as a function of δmax /Emax for cost function C3 (·), and battery sizes of (a) 6 kWh, (b) 12 kWh, and (c) 24 kWh.

different algorithms were evaluated on quantized values with a quantization step ranging from 0.125 kW to 0.5 kW depending on the chosen settings. To begin with, in Fig. 4 we show the cost improvement as a function of δmax /Emax , where Emax was set to 6kWh, 12kWh and 24kWh in the left, center and right graph, respectively. Cost function is set to C3 (·). As expected, AOS always provides the best performance, followed by DPM. DPI shows a small loss with respect to DPM, while ST provides the worst results, suffering from the constraints on the battery capacity and limited charge/discharge rates. Curves are clearly separated for high values of δmax , while for small δmax the narrow available range for δ(t) makes the four algorithms perform very similarly. We observe that, unexpectedly, the performance of ST deteriorates for large δmax . This effect is due to the correlations in the power demand L(t), for which high power requests are likely to occur in bursts. In this case, the ST policy will discharge the battery to fully satisfy the first request, leaving a residual charge in the battery that is not sufficient to fully satisfy the following requests. The resulting cost is much higher than that yielded by a strategy that partially satisfies each request in the burst, as typically performed by statistical algorithms as DPM and DPI. To support our hypothesis, we also verified that with non-correlated power request processes L(t), the performance decrease of ST for high δmax vanishes, though the result is not reported here for space constraints. In Fig. 5 we compare the performance achieved by the four algorithms as a function of Emax , for fixed δmax = 41 Emax , and with each of the three cost functions of Fig. 3. We see that performance linearly increases with the battery size Emax for small Emax , then it saturates to a maximum achievable improvement. Moreover, the figure provides evidence that the effective gain of all algorithms is strongly related to the cost function. A closer look reveals that all algorithms achieve

c 2013 IEEE THE FINAL VERSION OF THIS MANUSCRIPT APPEARED IN THE PROCEEDINGS OF THE IEEE ICC 2013, 9-13 JUNE 2013, BUDAPEST, HUNGARY. 5

γ

[%] 60

(c)

γ

AOS DPM DPI ST

[%] 60

50

(a)

70

C2 (·)

π

[%]

50

π

L2 −`

[kW]

60

50

(b)

10

γ

5

20

[kW]

10

C3 (·) 0

40

−e 0

40

30

−10

p(`)−e

C1 (·)

20



−10

20

−20

10

10

L1 −` 

10



L1 −` −15



20

30



30

L2 −`

p(`)−e

−5



40



(b) 70



(a) 70

−30

0

50

100

Emax [kWh]

0

50

100

0

Emax [kWh]

Fig. 5. Cost improvement (in percentage) for δmax = functions: (a) C1 (·), (b) C2 (·) and (c) C3 (·).

50

100

Emax [kWh] 1 E 4 max

−20 0

10

20

30

40

0

e [kWh]

δmin 10

20

30

40

e [kWh]

and cost

Fig. 6. Optimal policy π(e, `) with Emax = δmax = 48 kWh and ηc = ηd = 1 as a function of e for: (a) ` = 36 kW, (b) ` = 48 kW.

larger cost saving with cost function C2 , which also yields the larger performance gap between ST and DP algorithms. The reason is that the more fragmentary and steepish the cost function, the larger the potential benefit that a clever energy storage strategy can bring about. The above simulations highlight three main aspects: 1) ST algorithm is not optimal under realistic battery and power request models; 2) DPM does bring some performance gain over DPI because it better captures the correlation in the power request process L(t); 3) DPM may be very close to the AOS upper bound with realistic power request process. As a conclusion, DPM is the best candidate also because it does not suffer of performance deterioration for high charge/discharge rates (as seen in Fig. 4). However, a complexity issue arises in this respect, since a considerable computational effort may be required to determine the optimal strategy (but this can be made off-line). In fact, by equation (7), it can be seen that the computational complexity of the DPM solution can be expressed as NL NE (NL + NE ), where NE and NL denote, respectively the number of quantization levels for energy levels and for the power request process {L(t)}. More importantly, a wide memory storage is needed to implement DPM in practice. Fortunately, the optimal strategy π(e, `) has a very simple structure that can be described by very few parameters. We discuss it in detail for the case of cost function C3 with ηc = ηd = 1. Let L1 = 20 kW and L2 = 41 kW denote the two break points of cost function C3 in Fig. 3, respectively. Then, the ideal optimal strategy assumes the form illustrated in Fig. 6a, namely a two-level strategy linked by a linear connection with − π4 slope. The two levels are determined by the cost function break points, and are L1 − ` and L2 − `, respectively. Then the ideal strategy can be expressed as    π ˜ (e, `) = max L1 − `, min p(`) − e, L2 − ` (11)

dependence of `. Whenever this ideal strategy falls outside the allowable range A(e) = [A+ (e), A− (e)], then clipping must be performed in the form    π(e, `) = max A− (e), min π ˜ (e, `), A+ (e) (12)

where p(`) determines the position of the linear connection in

as illustrated in the example of Fig. 6b. Therefore, a simple map p(`) is sufficient to store the entire π(e, `), which is subsequently evaluated by only the use of equations (11) and (12). In the general case where a piecewise linear and convex cost function has an arbitrary number of breakpoints, B, and where efficiencies are different, ηc 6= 1/ηd , the strategy turns out to be multilevel, but still with linear connections at − π4 slope. A somehow similar behavior was proved in [7] in the linear case only with efficiency loss ηc , ηd < 1, but can be generalized. We leave to future work details and proof of this generalization. R EFERENCES [1] S. C. King, “The economics of real-time and time-of-use pricing for residential consumers,” American Energy Institute, Tech. Rep., 2001. [2] M. Albadi and E. F. El-Saadany, “A summary of demand response in electricity markets,” Electric Power Systems Research, vol. 78, no. 11, pp. 1989–1996, 2008. [3] H. Allcott, “Rethinking real-time electricity pricing,” Resource and Energy Economics, vol. 33, no. 4, pp. 820–842, 2011. [4] G. Barbose and C. Goldman, “A survey of utility experience with real time pricing,” Berkeley National Laboratory, Tech. Rep., 2004. [5] M. Lijensen, “The real-time price elasticity of electricity,” Energy Economics, vol. 29, no. 2, pp. 249,258, 2007. [6] I. Koutsopoulos, V. Hatzi, and L. Tassiulas, “Optimal energy storage control policies for the smart power grid,” in Proc. of Second Int. Conf. on Smart Grid Commun., Brussels (B), Oct. 2011. [7] P. Van de Ven, N. Hegde, L. Massoulie, and T. Salonidis, “Optimal control of residential energy storage under price fluctuations,” in Proc. of IARIA ENERGY 2011: The First International Conference on Smart Grids, Green Communications and IT Energy-aware Technologies, Venice (I), May 2011, pp. 159–162. [8] I. Richardson, M. Thomson, D. Infield, and C. Clifford, “Domestic electricity use: A high-resolution energy demand model,” Energy and Buildings, vol. 42, no. 10, pp. 1878 – 1887, 2010. [9] D. P. Bertsekas, Dynamic Programming and Optimal Control, 3rd ed. Athena Scientific, 2007.