2013 First International Conference on Artificial Intelligence, Modelling & Simulation

A Simulator for Intelligent Energy Demand Side Management

Glenn Platt

Ying Guo

Division of Energy Technology, CSIRO Newcastle, Australia [email protected]

Computational Informatics, CSIRO Sydney, Australia [email protected] implement. This can be achieved through Demand Side Management or DSM, which allows for energy to be managed at the point of consumption. Many demand management technologies are under development to allow for monitoring and/or automated control of devices [3]. “Smart devices” are able to be controlled remotely by the customer or energy retailer to decrease or postpone electricity consumption during certain time periods. Trials are currently being conducted to measure the success of smart devices deployed across the electric grid, which can receive control signals from utilities. Such technology can improve energy conservation, efficiency and reliability of the energy service provided by utilities. The traditional method of conducting case studies through deployment trials is very inefficient in terms of study time and cost. To achieve results with sufficient generality, the case studies need to cover a long period, such as at least one summer. They also need to cover a reasonable number and diversity of households, and the hardware and software for such studies can be very expensive. Australia’s Commonwealth Scientific and Industrial Research Organization (CSIRO) has developed a simulation environment to model different automated DSM strategies and their effect on the environment, economics and human satisfaction [4], [5]. The utility simulation tool (UST) assumes smart devices are installed in up to one million homes. Each household is modeled as a decision-maker within the environment. The UST can then forecast changes to each household’s energy consumption and satisfaction level. It can also analyze the global performance of largescale DSM deployments, in terms of the level and firmness of demand response alongside householder satisfaction. The UST also incorporates dynamic conditions such as weather and generator capabilities, which can affect the performance and fidelity of demand response. Household comfort levels are also factored in so as to project the acceptance and uptake of DSM strategies. This is one of the very first DSM simulators that take into account both technical and human behavior aspects. In this paper, we outline the functionality of the UST and the household model, and provide examples of the functionality and benefits of this tool. .

Abstract—Demand Side Management or DSM refers to the reduction or postponement of energy consumption. Current DSM technology can now provide automated off-site control of domestic and industrial devices. Many questions arise in regards to controlling a potentially large proportion of the population's electricity: To what level can we reduce demand? What incentives could retailers offer customers? How do we ensure consumers are satisfied? Previous trials of DSM control techniques have had various levels of success in reducing demand and in changing the consumption habits of individuals over time. The main criticism of existing automated control techniques is that they do not account for customer satisfaction and therefore do not survive in the long term. We propose a novel automated machine learning approach that incorporates customer satisfaction into automated demand reduction, satisfying both customers and retailers. Through a simulation of 200,000 households equipped with automated demand control, we conduct experiments measuring electricity levels alongside population satisfaction levels under different energy control policies. We illustrate that significant energy and cost savings can be achieved without compromising consumer satisfaction. Keywords- demand consumer satisfaction; learning

I.

side management; smart meters machine learning; reinforcement

INTRODUCTION

The gap between normal and peak electricity demand is growing wider as energy requirements grow exponentially with population increase and technological advance [1]. In the residential sector, there are typically two peaks of electricity demand that occur in the morning and at night. These peaks correspond with our daily routines for washing, drying, cooking, etc. This means that around 20 percent of the time, electricity generation and distribution infrastructure is operating at close to its capacity, and the rest of the time it is operating much less than rated capacity, giving poor utilization of these assets. Therefore, we are not using our electricity infrastructure as efficiently as we could be. Long-term investment is needed to ensure that generation and distribution infrastructure can provide adequate supply of electricity during high demand periods. This peak demand problem can be solved in one of two ways: 1) Increase the supply of electricity or 2) Reduce or defer the consumption of electricity within specified time windows [2]. Increasing electricity supply requires heavy investment in capital and time. On the other hand, reducing or deferring electricity demand requires relatively low investment and is fast to 978-1-4799-3251-1/13 $31.00 © 2013 IEEE DOI 10.1109/AIMS.2013.64

II.

SIMULATION ENVIRONMENT

We develop a simulation environment to model different automated DSM programs and their effect on the environment, economics and human satisfaction. We model

303 309

households as the decision-makers within the global environment. Each household has a set of appliances or devices that determine the level of electricity consumed over time. Using electricity incurs a cost related to the global electricity usage at that point. The level of electricity usage affects the price of electricity, which in turn affects consumer decisions. Global environmental conditions such as weather fluctuate over time and can affect electricity consumption and consumer happiness.

Our simulation model can be divided into three separate components which handle electricity price, household satisfaction and energy consumption. In this model, modules are connected in a circular fashion where demand drives cost affecting household satisfaction which changes demand. Figure 1 illustrates the dependencies between model components.

Figure 1. Dependencies between demand, price and customer satisfaction.

Figure 2. Australian energy flow chart.

Our simulator is specifically targeted at modeling the energy usage and consumer habits of Australian residential customers within the New South Wales (NSW) state region. We therefore use data specific to the NSW region where possible, otherwise we utilize Australian-level statistical information. The simulator can be easily implemented to other area, as long as the local consumer’s energy relevant information can be implemented in to the modeling process.

flow under management of AEMO. Within such flow structure, wholesale trading in electricity is conducted under a spot market where supply and demand are instantaneously matched in real-time through a centrally-coordinated dispatch process. Generators offer to supply the market with specific amounts of electricity at particular prices. From all offers submitted, AEMO determines which generators are to produce electricity by meeting prevailing demand in the most cost-efficient way. AEMO then dispatches these generators into production. A dispatch price is determined every five minutes, and six dispatch prices are averaged every half-hour to determine the spot price for each trading interval for each of the regions of the NEM. AEMO also sets a maximum spot price of $13 100 per Megawatt hour. This is the maximum price at which generators can bid into the market. During most times of the year, the spot price is low (less than $60 per Megawatt hour), but every now and then,

A.

Australian Energy Network Model Firstly, the Australian Energy Network is simulated. In Australia, the Australian Energy Market Operator (AEMO, previously called the National Electricity Market Management Company Limited) was established in 1996 to administer and manage the National Electricity Market (NEM), to develop the market and continually improve its efficiency [6]. Figure 2 shows the typical Australian energy

310 304

such smart devices. We model two of the most popular ones as described below. Direct Load Control (DLC) has one or several “circuits” that allow different categories of household appliances to be switched off by the utility at times of peak demand. Switching may be through a physical circuit, interrupting the flow of electricity, or through a broadcast communications method that activates a local switch at the appliance or circuit board. Customers receive a discount on their electricity bill, or another kind of reward, that may be based on which appliances are signed up to different circuits, how often those circuits are switched, or on the achieved system outcome which of course is what generates value for the utility. This kind of program has been used to a limited extent for hot-water systems in Australia and is now being trialed on a wider range of appliances such as air conditioners by many utilities. Price-Based Control (PBC) exposes customers to a varying retail price that is increased sharply at times when the utility would like a reduction in demand. Customers get advance notice of high prices by one or several means and can then choose which appliance settings to change, if any. Typically the price can be at several discrete levels, for example, low and medium rates that are less than the average retail electricity price, and high and critical rates that are more. The varying price may approximate the wholesale market price to some extent, or it may be based on local network loading not reflected in wholesale pricing. Customers receive rewards in a similar manner to DLC programs.

it can also be very high (near or at the maximum price), when there is a peak demand. This occurs when energy supply is under extreme pressure, such as when there is extreme high temperature on a hot summer day. Electricity is not currently economically storable, and being a volatile commodity, production is subject to rigid, short-term capacity constraints. Since demand is highly variable, this means there will be times when there is plenty of capacity and the only incremental costs of producing electricity are fuel, operating and maintenance costs. At other times, the capacity constraint will be binding, causing the incremental cost to increase greatly and market prices to rise. B. Local Energy Consumption at the Household Level Given a population's electricity usage for a particular region, we wish to derive individual household energy usage over time. The electricity fluctuations of an individual household will allow us to model the effect of switching on or off particular devices which will propagate to energy consumption trends of the population. In order to scale the energy demand of many households down to the level of an individual household, we first need to determine the most indicative factors which affect electricity consumption. There are many factors correlated with energy usage, the most significant being the number of adults residing within the premises, their age, their income and whether they have children or not [7]. Ironmonger et al conducted a study using Australian National Energy Survey data (7405 Australian households in 1988-89) to generate an energy usage model. We derive the energy usage of households by first categorising each one into the three age-based categories: young, mixed and older. We then apply a category-specific formula to calculate annual energy usage for each household. The formula for annual energy usage is derived by fitting a quadratic regression model to each household type and a linear regression model to household income. The fitted functions are:

B. Human Behaviour Modelling In essence, both price based control and direct load control describe an “opt-in” control program, where the user decides whether to trust external control fully (DLC) or to modify their consumption behavior themselves (PBC). Trials for both of these programs have had limited success. DLC programs have often shown poor customer satisfaction because the customer has little input into electricity restriction directives. Customers are typically forced to accept the decision from a utility or retailer to turn devices off in return for an incentive. Once the DLC program is in place and the customer has committed to the contract, there is little that can be done to modify or tailor directives to suit individuals, except for opting-out of the entire program. With Price-Based Control programs, the user has more power to decide what appliances are to be switched on or off. Hence, this program typically exhibits higher satisfaction levels for participants. However, in terms of electricity consumption reduction, trials of price based control programs typically show that they are not durable in the long term. Reducing power consumption is successful within the first month but this drops as novelty wears off and old habits creep back into place. People are inconvenienced in switching devices off, since they must make a conscious decision and the immediate reward is not worth the effort. Both DLC and PBC programs regularly fail because of one or more problems with consumer satisfaction, ease of use or long term sustainability. What is the underlying reason

E young = 356.2  122.6 u n  18.7 u n 2  1.77 u i (1) E mixed = 332.4  71.8 u n  7.1 u n 2  3.47 u i (2) E older = 327.6  84.1 u n  6.6 u n 2  5.14 u i (3) where n is the number of adults, i is the household income per year ($AUD) and E is the energy expenditure per annum (kW). The above formulae result in a value which indicates the total energy usage over the period of one year. In order to simulate events on a minute by minute basis, we translate these values into a weight for each household. We can then create a fluctuating time series within the simulator by applying this weight to the average household energy consumption. III.

INTELLIGENT DEMAND SIDE MANAGEMENT STRATEGIES

A. Household Level Modelling With the ability to control devices remotely, different programs have been proposed to allow for fair operation of

311 305

for these problems occurring? Human factors are not taken into account to the level that is required. In order to track the satisfaction of consumers, we first need to find a way to quantifiably measure this. Such a problem is very difficult how does one translate human comfort into a number on a scale? Statistics on participation rates and post-program feedback ratings are useful but do not encapsulate the level of consumer satisfaction over time during the program. Surveys can be issued throughout the duration of trials but provide another added inconvenience to the user. A more standardized way of measuring the success or failure of these programs is needed so that they can be compared and ranked. We propose a new way to track consumer satisfaction throughout the duration of automated DSM strategies. For the simulator, we mainly need to judge whether households feel content with the direct load control signal (disturbance) from the energy supplier. Since we elect air conditioners as the controllable loads, internal temperature is taken as the determining factor for user satisfaction. According to the ASHRAE standard 55-2010 [8], people’s dissatisfaction can be simulated as a polynomial. The ASHRAE thermal sensation voting scale is defined as a seven point scale from hot to cold. Since we do not have such statistical voting data, we define a person’s “instant dissatisfaction” as a combination of polynomial functions:

) (t ) O1 T (t )  Tsc

M1

M

 O2 T (t )  Tsh 2 ,

the peak demand. The goal of a broker is to retrieve up-todate information on the market and demand requirements, make dynamic, informed decisions on local energy consumption limits and propagate this information down to the local level. The information passed to a local group is referred to as a cap, and is the upper limit of energy consumption allocated to a particular group. The sequence of events for deciding a cap in the broker is as follows. Firstly, the broker communicates with external parties to obtain knowledge of current and historical data, such as the latest market price, regional demand, local weather, etc. Secondly, the broker communicates with the local groups to gain knowledge of local energy demand requirements. These local energy demand requirements are expressed as two sets of information: 1) the minimal achievable consumption of the group and 2) the unconstrained (default) consumption of the group. Here minimal is defined as the total minimal consumption over an entire future market cycle interval (e.g., five minute interval). The unconstrained (default) consumption of the group refers to the total consumption of the group when households operate under normal conditions without external influence. Lastly, the broker calculates the cap on the next market cycle interval for the group level using all information available to it. The cap calculated by the broker lies inclusively between the values of the constrained and unconstrained plans sent from the group.

(4)

where

O1

0 when T (t ) d Tsc , O2

0 when T (t ) t Tsh .

Energy Demand (MW)

Here Tsc is the higher bound of the comfortable temperature zone, Tsh the lower bound of the comfortable temperature zone, M1 and M 2 are powers applied to the

temperature difference. Also, O1 and O2 are the weights for the dissatisfaction caused by an internal temperature that is either too hot or too cold, due to the direct control of the air conditioner by the energy supplier. It is observed that the human body has a memory of dissatisfaction, hence we define the dissatisfaction function as: (5) *(t ) w*(t  1)  ) (t ), where *(0) ) (0) where the discount-rate parameter w has a range of 0 to 1. It shows the effect of previous dissatisfaction on the current time step. Because different people may have different satisfaction levels for identical temperatures, the value of can be adjusted by tuning the values of M1 , M 2 , O1 and O2 .

Before Peak Shaving

After Peak Shaving Time of Day (24hr)

Figure 3. Optimsation on Cost ($), energy demand (MW), and consumer dissatisfaction level.

D. System Optimisation The broker needs to define a cap which not only reduces energy costs, but also avoids instability within the system. Meanwhile, the local group and each household need to balance cost and dissatisfaction. The main goal now is to find an algorithm that the broker can use to set a nearoptimal cap for the group, and the local controller can use to reduce the cost and dissatisfaction. The judgment of the “optimal” solution can be measured based upon three important elements (see Figure 3): x Cost: the global cost of utilizing energy; x System stability: the energy demand distribution along the time dimension. x Local dissatisfaction: consumer satisfaction throughout the duration of automated DSM strategies. In this optimization problem, there is insufficient information to apply a supervised learning methodology. The

So we can set different dissatisfaction function *n (t ) for household n according to equations (4) and (5). Note that the dissatisfaction function can be defined in other formats according to different energy supplier control strategies or situations. C. Broker Level Modelling When the electricity market operator (AEMO) makes a decision, their highest priority is power system reliability. To improve reliability of the power system and avoid the maximum spot price, one approach is to intelligently reduce

312 306

market changes its behaviour due to many direct and indirect factors such as local temperature, weather condition, market spot price and predicted demand. It is therefore necessary to use a learning methodology which allows the broker to learn its behaviour online based on feedback from the environment. We choose two machine learning algorithms, one is a genetic algorithm for local level optimisation between the household energy consumption and dissatisfaction level. The other one is a reinforcement learning (RL) approach, Q-learning, to allow the broker to learn from experience [9][10]. Behavioral learning is adaptive with time, allowing RL to cope with control of a dynamic system, such as energy demand. Adopting behavioral learning provides the added advantage that there is little need for the broker to know about the system. The broker can learn how to set the cap through trialand-error interactions with the environment [11]. IV.

simulation results are plotted. If the mouse points to one of the houses (as the one be circulated in Figure 3), the details of that household’s status can be shown in the result section. If no house is chosen, the result figures show the total or average status of the whole community. The user can choose to save the running results into separate files. The result figures clearly show daily and annual cycles. Consumption and satisfaction levels of 1 000 000 households are computed at every simulation step. In the following experiments, AEMO datasets were used that give price and demand values from the January 2006 period. There were several price peaks during this month for the Australian state of NSW, where the highest value was $529.95 on 23rd January 2006. In fact, the highest demand (12674.6 MWh) also occurred on the same day. B. Local Level Optimisation using Genetic Algorithm

EXPERIMENTAL RESULTS

In this section, we will firstly show the simulation tool interface. Then two optimization results of DSM are listed, one for household level, and one for global level. A. Utility Simulation Tool Interface

Figure 5. The average dissatisfaction level

Figure 4. The Utility Simulation Tool GUI.

In order to facilitate the understanding of potential consequences of various DLC actions, in terms of both the energy consumption and customers’ satisfaction levels, we design the GUI of the UST as in Figure 4. In the top left corner one can set up parameters and execute the simulation. These parameters include which DSM strategy to use; the ranges of temperature set-points; and the percentage of total households those are willing to be externally controlled. In the bottom left corner, current time and current cost can be displayed. As shown in the middle of Figure 4, a Sydney map is imported. One hundred small houses are plotted in the map, where each house stands for one percent of the simulated households. The color of each house stands for the energy consumption, where lighter color stands for less energy consumption and darker color for higher energy consumption. On the right hand side, some of the key

Figure 6. Comparison of power consumed by air conditioners under the optimal control strategy (solid line) and the one without energy supplier control (dashed line).

The genetic algorithm is trained using the recorded temperature and NEM energy price datasets. The parameters in equations (4) and (5), such as M1 , M 2 , O1 , O2 and w are optimized via the learning process. Then the optimized local level demand management strategy can be implemented on different dates. Let us now compare the system performance between “without any control from broker” to “optimal control from broker”. Figure 5 illustrates the average household dissatisfaction levels, which shows four different results:

313 307

ವ red line for no global control where people are roughly satisfied all the time; ವ black line for totally turning off all air conditioners when the energy cost is exceedingly high, hence the dissatisfaction level is high when this approach is chosen; ವ green line for the optimized management result, where the dissatisfaction level is nearly zero except during the extreme high energy price period. Figure 6, on the other hand, shows the average energy consumption using the default management strategy versus the optimized management strategy. The testing results over January show that the optimized local level DSM is much better than default strategy. The energy saving is about 50% for the whole summer.

V. CONCLUSION The electricity market is a dynamic system, where demand and price can fluctuate dramatically. Demand management is a recent attempt to control fluctuations in energy requirements at the appliance level, hence reducing the cost of energy provision. Smart devices can monitor and control the energy consumption, essentially delaying their energy usage when the demand for electricity is high. This architecture provides benefits to multiple stakeholders, including monetary relief to consumers, as well as relief to the energy network infrastructure. In order to achieve lower energy cost without compromising system stability and human satisfaction, we propose a learning approach to optimize the system-level goals. Using this learning approach, we are able to build a dual-purpose reward function on increasing system stability as well as reducing energy cost, and maintain the dissatisfaction level at a low level. The method we introduce uses only a small state space for both the decision and reward matrix but still produces effective outcomes. For future work, we wish to explore alternative methods for setting system-level goals. A reduction in parameter space and/or in training time whilst also providing system stability and cost reduction would be ideal. REFERENCES

Figure 7. Cap setting where optimization is based on balancing cost and system stability. Solid line – cap; dash line -- market price.

[1] EIA, “International Energy Outlook 2009,” 2009. [2] S. Borenstein, M. Jaske, and A. Rosenfeld, “Dynamic Pricing, Advanced Metering, and Demand Response in Electricity Markets,” 2002. [3] R. Rankin, P. G. Rousseau, and M. van Eldik, “Demand side management for commercial buildings using an inline heat pump water heating methodology,” Energy Conversion and Management, vol. 45, no. 9–10, pp. 1553–1563, Jun. 2004. [4] Y. Guo, R. L. R. Li, G. Poulton, and A. Zeman, “A Simulator for Self Adaptive Energy Demand Management,” in 2008 Second IEEE International Conference on SelfAdaptive and SelfOrganizing Systems, Ieee, 2008, pp. 64–73. [5] Y. Guo, A. Zeman, and R. Li, “Utility Simulation Tool For Automated Energy Demand Side Management,” pp. 37–44. [6] R. C. Bansal, Z. Y. Dong, K. N. Hasan, N. H. Radzi, and Z. Lu, “Overview of the Australian national electricity market transmission use of system charges for integrating renewable generation to existing grid,” IET Generation, Transmission & Distribution, vol. 6, no. 9. p. 863, 2012. [7] B. Erbas, D. S. Ironmonger, and C. K. Aitken, “Economies of scale in energy use in adult-only households,” Energy Economics, vol. 17, no. 4. pp. 301–310, 1995. [8] Ashrae, “ANSI/ASHRAE 55:2010 Thermal Environmental Conditions for Human Occupancy,” Ashrae Standard, vol. 2004. p. 30, 2012. [9] R. S. Sutton and A. G. Barto, “Reinforcement learning: an introduction.,” IEEE transactions on neural networks / a publication of the IEEE Neural Networks Council, vol. 9, no. 5, p. 1054, 1998. [10] P. Dayan and C. J. C. H. Watkins, “Q-learning,” Machine Learning, vol. 8, no. 3, pp. 279–292, 1992. [11] Y. Guo, A. Zeman, and R. Li, “A Reinforcement Learning Approach to Setting Multi-Objective Goals for Energy Demand Management,” International Journal of Agent Technologies and Systems, vol. 1, no. 2, pp. 55–70, 2009.

C. System Optimisation using Reinforcement Learning At the system level, the broker learns how to set the demand cap via trial-and-error interactions with the environment. The system optimization matrix is learnt using the reinforcement learning approach as explained in the previous section. Because the reward matrix is learnt online, the reward function is crucial to the optimization solution. With different reward functions, different cap optimization results were achieved. For instance, if cost is given too great a weighting in the optimization, the system may lose stability. The learn strategy of the broker can achieves lower energy cost without compromising system stability and human satisfaction. The test results illustrated in Figure 7 show the system negotiating the balance between demand cost and system stability. The red line is the real market energy price, and the blue line is the optimized cap for energy consumption. Clearly, the low cap matches the high price very well. When low market prices occurred, the cap is quickly released to 90% of demand requested, avoiding future demand oscillation. Hence, the real energy consumption is lower than the unconstrained consumption while the system stability is achieved as well. These experiments demonstrate the ability for the algorithm to be applied to a real-time environment where direct feedback is implemented.

314 308