IEEE TRANSACTIONS ON INDUSTRIAL INFORMATICS, VOL. 6, NO. 4, NOVEMBER 2010

503

Runtime Allocation of Optional Control Jobs to a Set of CAN-Based Networked Control Systems Pau Martí, Member, IEEE, Antonio Camacho, Manel Velasco, Member, IEEE, and Mohamed El Mongi Ben Gaid

Abstract—The Controller Area Network (CAN) provides the basis for many cost-effective distributed embedded systems. In this paper, we present a novel approach to networked control systems (NCS) analysis and design that provides increased control performance for a set of control loops that share a single CAN network. This is achieved by enabling the following functionality for each control loop: first, standard periodic messaging is guaranteed to ensure stability, and second, non-periodic additional messaging is added whenever bandwidth is available in such a way that the aggregated control performance for all control loops is improved. To validate the presented approach, a proof-of-concept implementation is presented, and the extensive experimental results show the type of benefits that can be achieved as well as the resulting behavior depending on several key parameters. Moreover, it demonstrates that the approach can be implemented in practice. Index Terms—Controller area network (CAN), control performance maximization, linear quadratic (LQR) control, networked control systems (NCS).

I. INTRODUCTION HE controller area network (CAN) [1] protocol was originally designed for the automotive industry, but has also become a popular bus in industrial automation as well as other applications. CAN provides the basis for many cost-effective distributed embedded applications, e.g., [2], [3], and its properties and functionalities such as reliability, dependability, or clock synchronization, are being constantly enhanced [4]–[8]. In a parallel track, networked control system (NCS) [9], i.e., control loops closed over communication networks where sensors, controllers and actuators are physically distributed and exchange control data through the network, are gaining increased attention in many control application areas including motors, suspension systems, converters, or industrial manipulators [10]–[14], while the underlying required control theory is starting to offer mature and methodological results [15]. This paper, taking advantage of structural properties of CAN, presents a novel approach to CAN-based NCS analysis and design that provides increased control performance for a set of

T

Manuscript received October 26, 2009; revised March 19, 2010, July 19, 2010; accepted August 24, 2010. Date of publication September 23, 2010; date of current version November 05, 2010. Paper no. TII-09-10-0268. P. Martí, A. Camacho, and M. Velasco are with the Automatic Control Department, Technical University of Catalonia, Pau Gargallo 5, 08028 Barcelona, Spain (e-mail: [email protected]; [email protected]; [email protected]). M. E. M. Ben Gaid is with the Technology, Computer Science, and Applied Mathematics Division, IFP New Energy, 92852 Rueil-Malmaison, France (e-mail: [email protected]). Digital Object Identifier 10.1109/TII.2010.2072961

control loops that share a single CAN broadcast domain. The baseline idea of the approach is to use the available spare bandwidth for applying as many control actions as possible in such a way that the aggregated control performance is improved, and this is achieved by transmitting specific additional messages whenever the bus is idle in an instantaneous manner, without requiring to know a priori the fraction of available bandwidth. In addition, when spare bandwidth can be consumed, each control action accommodated in the bus is chosen following a procedure that, taking into account the controlled plants’ dynamics, ensures the best control performance increase among all possible choices. A. Contribution and Summary The approach presented in this paper enables the following functionality for each networked control loop. First, periodic execution of each control algorithm is imposed to guarantee stability and minimum performance guarantees. Hence, each guaranteed control loop operation requires sending the corresponding sensor (sensor to controller) and controller (controller to actuator) messages periodically. Second, additional non-periodic executions of each control algorithm are promoted in order to improve the aggregated control performance as long as bandwidth is available for the corresponding required additional non-periodic messages. Enabling this functionality requires solving two problems. The first problem is to develop a suitable policy for choosing to which control loop the additional messaging should be allocated to. The goal is to make an efficient use of bus idle times among those control loops contending for the bus in order to improve the aggregated control performance. This problem has a constraint that must be considered: its solution should be easy to implement in terms of complexity. This problem is referred as the allocation problem. The second problem concerns the design of the control algorithm for each control loop. The design should guarantee that the more control updates can be executed, the better will be the control performance. This problem is referred as the control design problem. The allocation problem is addressed by observing that certain functions such as MIN or MAX (minimum and maximum value, respectively) can be efficiently implemented in CAN [16]. Looking at CAN, the key idea is to encode quantities in sensor messages identifiers (and to achieve contention for the medium) in such a way that, after the operation of the bitwise arbitration phase, the desired function is accomplished. Using this approach, functions can be computed efficiently, without increasing the number of messages, and obtaining the solution with a time-complexity that does not depend on the number

1551-3203/$26.00 © 2010 IEEE

504

IEEE TRANSACTIONS ON INDUSTRIAL INFORMATICS, VOL. 6, NO. 4, NOVEMBER 2010

of nodes (i.e., number of sensors). Using this principle, and under the optimal control formalism, the paper shows that the allocation problem can be formulated as a MAX problem that is then efficiently solved by the CAN bitwise arbitration. In particular, the message encoding that is required for implementing the allocation policy is designed for CAN 2.0 B messages, that is, for messages with 29-bit identifier. In addition, the type of computations needed in sensor nodes for coding the required quantities in message identifiers are analyzed in terms of computation and memory demand. It has to be highlighted that the bitwise-based solution that the paper presents increases the number of needed message identifiers in contrast of current practices of CAN management. The control design problem is addressed using two alternative approaches based on linear quadratic control techniques, one borrowed from the literature [17], while the other is developed in this paper. Both ensure different levels of achievable control performance improvement while requiring different memory demands at the controller node. To validate the presented approach, a proof-of-concept implementation is presented, consisting on a pool of three unstable electronic circuits controlled over a CAN network. The experimental results show that the presented approach is able to increase control performance as soon as bandwidth is available. In addition, the effect of different key parameters of the presented approach is evaluated in terms of control performance, such as control design parameters or different network load conditions. Finally, comparisons with previous relevant work are also reported. The rest of this paper is organized as follows. Section II discusses the related work. Section III formalizes the NCS model and the allocation problem. Section IV presents the solution to the allocation problem. Section V presents the solution to the control design problem. Section VI analyzes the computational and memory demands at sensor nodes for implementing the allocation policy. Section VII illustrates some of the introduced concepts using a simple NCS example. Section VIII describes the implementation setup and summarizes the more relevant experimental results. Section IX concludes this paper. II. STATE-OF-THE-ART The most common design and implementation approach for NCS consists in the periodic execution of the control algorithm, which implies that messages from sensors to controllers and from controllers to actuators are periodic [18]. The benefit of this approach is that each control loop stability can be easily guaranteed at the expenses of making a static use of the communication bandwidth in the sense that the given messages periodicity is imposed from the control design stage regardless of the current load in the network and/or changes in the systems that are being controlled. To overcome this periodicity limitation, two new tendencies for the analysis and design of NCS can be identified. The first one is to apply rate adaptation techniques where the period is selected according to the controlled system dynamics and/or to the bandwidth conditions, e.g., [11] and [19]. In the same line, many rate adaptation results targeting microprocessor-based

systems enabled with real-time technology, e.g., [20], [21], and references therein, can be also adapted to the network setting, e.g., [22]. The main goal of these approaches is to improve the aggregated control performance for the set of control loops by efficiently using all the communication bandwidth. The second tendency is to apply event-based sampling techniques which produce non-periodic executions of the control algorithm, and therefore, non-periodic messaging in the network, e.g., [23]–[25]. The main goal of these approaches is to minimize the bandwidth utilization while still guaranteeing stability and acceptable control performance. The approach presented in this paper shares the same goal of the rate adaptation techniques, that is, it aims at efficiently using the available bandwidth to improve the aggregated control performance. However, it overcomes the main drawbacks of rate adaptation techniques. In general, in rate adaptation techniques, any share of available bandwidth for applying the rate adaptation has to be known a priori and it also has to be guaranteed over a period of time. The approach presented in this paper breaks this limitation. Additional non-periodic control messages will use the bus whenever it is idle, instantaneously. In addition, rate adaptation techniques often require solving at runtime an optimization problem for selecting sampling periods according to the plants’ dynamics. However, such optimization problems do not have a closed analytical solution in the general case, which prevents their implementation due to the runtime overhead. It is worth noting that recent results in rate adaptation techniques [26] indicate that the best metric to consider for online resource allocation is to use predictions of the expected plant dynamics. However, this metric requires knowing the activation pattern of all future control jobs, which can not be known in the scenario considered in this paper. Hence, if such predictions can not be applied, alternative metrics must be considered. By developing a metric that accounts only for those jobs that are guaranteed (the periodic ones) plus the possible benefit of executing an additional control job, the approach presented in this paper permits selecting which control loop must execute the additional control operation considering all controlled plants’ dynamics, and its implementation can be easily performed in CAN with bounded time complexity. In addition, the flexibility of the adopted solution of using the bitwise arbitration phase of CAN to solve the allocation problem could be also achieved using centralized protocols capable of providing some sort of rate adaptation, such as the flexible time-triggered paradigm applied to networks [27], [28], or the networked control system approach based on a token passing protocol [10]. Although centralized management introduces some undesired features, it also greatly simplifies global management, for example, eliminating the need for a specific type of medium access control protocol and making the approach much more general. Future work will consider porting the approach to flexible centralized protocols. The results in [29], [30] (further developed in [22]) present similarities with the approach presented in this paper. However, both approaches, named Max Error First. Try Once Discard (MEF-TOD) and Largest Error First (LEF), respectively, encoded in sensor message identifiers an heuristic quantity, the

MARTÍ et al.: RUNTIME ALLOCATION OF OPTIONAL CONTROL JOBS TO A SET OF CAN-BASED NETWORKED CONTROL SYSTEMS

plant error. Using this coding, MEF-TOD shows that global exponential stability under certain conditions can be achieved, while LEF focuses on control performance improvement. In [31], the analysis and design of similar dynamic contention protocols over CAN networks were improved (in particular, in terms of stability analysis conservativeness). The stability guarantees of the approach presented in the paper are given by the guaranteed periodic operation of each control loop. The message coding of the approach presented in this paper obeys control theoretical principles and it is shown to provide better control performance improvement than the LEF approach. The approach presented in [25] offers a high-level formulation/framework under which the allocation policy presented in this paper can be accommodated. However, they heuristically define an instantaneous metric for online resource allocation rather than a metric based on some sort of prediction derived applying control-theoretical principles, as it is done in this paper. In addition, they do not prove using real experiments the validity of their approach. Finally, performing a comparative analysis is not straightforward because both approaches target different goals in terms of specifications. The approach presented in this paper is also connected to multirate control techniques, e.g., [32] and [33]. In multirate control over networks, an outer loop mandates to sample at the sensor node at a low frequency to minimize sensor messages. Then, the inner loop at the controller (or actuator) node produces estimates whenever fresh sensor data is missing to generate actuation at a faster rate. The goal is to achieve the same performance that would be achieved by the faster loop using always fresh data. In a similar context, a dual-rate switching control approach was presented in [34], in which two sampling rates are used. A fast rate is specified to meet the control requirements. However, when the system is near stationarity, a low rate is used, allowing saving resources without significant degradation in control performance. Although the additional messages in the approach presented in this paper will be given a fast rate for implementation purposes, they are not guaranteed, and therefore, they are aperiodic and their transmission is not known a priory. On the contrary, the two sampling rates in [34] or the rates for the outer and inner loop in [32] and [33] are known a priori and periodic. The idea of dividing control messaging into two types of messages, guaranteed and additional (non-guaranteed) messages, is similar to the approach taken in [35] where the different jobs of a control task are classified into mandatory and optional in order to achieve a graceful degradation in a situation of processor overload. In that case, the controller is designed for a given rate in order to minimize the degradation, tolerating occasional sampling vacancies. The approach presented in this paper uses a different controller design methodology, with a different control goal, and targeting a different architecture, NCS. In particular, one of the design methodologies used in the approach presented in this paper can be found in [17], where the differences with respect to [35] are clearly stated. Finally, in a complementary area of research, several results have been presented to provide efficient and/or optimal scheduling of messages in the CAN network, e.g., [36]–[40]. However, they mainly focus on improving some metrics related to

505

Fig. 1. Networked control system scheme.

real-time timing constraints rather than improving control performance as the approach presented in this paper. However, it is worth noting that some of these metrics such as message latencies can have a direct impact on control performance. III. PROBLEM FORMULATION A. Networked Control System (NCS) Model The NCS considered in this paper is illustrated in Fig. 1. Several single input/single output control loops, each one formed by a sensor, a controller and an actuator implemented in physically separated nodes, share a single CAN bus (single broadcast domain) to exchange the control data required for each control loop operation. In addition, other nodes, represented by the load boxes, also use the CAN network to exchange other non-control data. For a given networked closed-loop system, a control job will denote the required operations for each plant update. Hence, each control job would basically require sending the sensor reading in the sensor message after sampling the plant, and sending the control signal in the control message after computing its value in the controller node using the information contained in the incoming sensor message. Thus, in terms of bandwidth utilization, each control job simplifies to sending two messages, the sensor and the control message. B. Plants and Cost Functions is described by a continEach control loop uous-time plant whose input is given by a linear discrete-time controller, and a quadratic cost function. To keep the notation simple, the loop index is dropped in the plant, controller, and cost descriptions below whenever not required. Each plant is described by a continuous-time linear system

(1) , and are the system, where input and output matrices, respectively, and where is the plant state, the system input (control signal), and the system output (plant response). Each control loop performance is measured by an infinite-horizon continuous-time quadratic cost function (2) where and are constant weighting matrices over the integration time. As example of notation, for the particular th control loop, . cost (2) is referred as

506

IEEE TRANSACTIONS ON INDUSTRIAL INFORMATICS, VOL. 6, NO. 4, NOVEMBER 2010

C. Mandatory and Optional Control Jobs Within each control loop, two types of control jobs can be distinguished depending on whether they are guaranteed or not. Each th control loop has a guaranteed constant sampling period , which means that at least, with a given periodicity, the plant is sampled and the control signal is computed and updated (and held constant until a next control job executes). These control jobs are called mandatory control jobs. Note that mandatory control jobs of different control loops can have different guaranteed sampling periods. However, for the sake of paper easy readability, the subindex for guaranteed sampling periods is avoided. In addition, each control loop can make use of additional samples between mandatory control jobs to further update the control signal. These samples are taken at a given (fast) rate, that for implementation purposes, is specified by a short sampling pe. Each control job using an additional sample is called riod an optional control job because its execution depends on the available bandwidth. If executed, a new control signal is computed and updated (and held constant until a next control job executes). Henceforth, it is assumed that all short sampling rates for optional control jobs for all control loops are the same, , and that the guaranteed sampling periods for mandatory control jobs for each ith control loop is a multiple of the short sampling period . The theory presented in this paper does not require these assumptions. However, its implementation mandates to define exact numbers, and different issues like achieving bus contention and/or treating network induced delays simplify when these assumptions are taken. Moreover, as it is further explained in Section IV-C, transmissions of samples, for mandatory or optional jobs, need to be synchronized for the allocation mechanism to work properly. The controller gain of mandatory control jobs is designed using periodic linear quadratic (LQR) control theory for the given period, which ensures stability and minimal acceptable performance. The design of gains for optional control jobs is discussed in Section V. For both type of control jobs, the system in (1) is given as follows. Let denote a control job input execution and let denote the time at which the -control job executes, that is, it samples the state, , and computes and updates the control signal, . For any -job, either mandatory or optional, the control signal is given by

trol jobs is of interest. Integrating (2) over intervals given by every two consecutive control jobs gives (4) where

and denotes two consecutive control jobs, denotes the cost during the time interval denotes the sampled state, and the updated control signal generated by the job, and where

(5) Noting that (see [41])

where

and reminding that the control signal is held constant over consecutive control jobs, the cost over two consecutive control jobs (5) in discrete form is (see [41])

(6) where (7) (8) (9) Cost (6) can be written in compact form as (10) where

(3) The controller gain in (3) will be further explained in Section V. As explicitly expressed in (3), the control signal for the -job is held constant until the next control job executes at time , which can be the next mandatory job or an additional optional control job executing after the -job and before the next mandatory job. D. Discrete Cost Considering the continuous cost function (2), the equivalent cost at discrete time instants given by the execution time of con-

(11) Again, as an example of notation, for the particular th control loop, cost (4) can be referred as . E. Short Notation This section introduces a short notation that drastically simplifies the paper readability. First of all, for a given th control as in loop, costs between consecutive control jobs and . Hence (6) or (10) will be denoted by (12)

MARTÍ et al.: RUNTIME ALLOCATION OF OPTIONAL CONTROL JOBS TO A SET OF CAN-BASED NETWORKED CONTROL SYSTEMS

507

Second, for a given th control loop, considering the time in, the cost achieved by executing only mandatory terval is denoted by jobs starting to execute at time (13) where denotes the time when the previous job execution took place (if any) before the execution time of the first mandatory . Hence, it may correspond to the exejob under evaluation, cution time of the previous mandatory control job, if , or it may correspond to the execution time of a pre, where can be the vious th optional control job, optional control jobs transmission instant of any of the between the previous mandatory control job and the one under evaluation. The parameter will determine which state for the first term of the summand (when ) has to be considered for evaluating this type of expressions. Note that when (initial condition in system initialization), then . F. Problem to Be Solved The NCS model assumed in this paper considers control loops, each one with a plant in the form of (1) whose control signal is given by (3) for both mandatory and optional control jobs (if possible), and whose control performance is evaluated via (4)–(6). For the control loops, at any time different than mandatory control jobs execution times, provided that bandwidth is available for executing an optional control job, the problem to be solved is to which control loop the optional control job should be allocated to, such that the overall accumulated cost (14) is minimized. Note that (14) accounts for both mandatory and optional control jobs. IV. OPTIONAL JOB ALLOCATION POLICY This section defines a suitable policy for allocating optional jobs to control loops. Then, it transforms this policy into a MAX problem for easy implementation in the CAN network. A. Minimum Cost Policy Solving the problem posed in Section III-F is not easy because the time at which optional control jobs will execute is not known a priori. It depends on the runtime available bandwidth. Considering this limitation, the allocation algorithm presented in this paper is based on decreasing the aggregated cost (14) by evaluating at run time if the execution of an optional control job each time bandwidth is available effectively decreases the cost. Hence, rather than providing an optimal solution, the policy guarantees that the cost is decreased. Specifically, from costs are evaluated. Each one correthe current time to sponds to the total aggregated cost that would be achieved if the optional control job would be allocated to a particular th control loop. From the costs, the minimum indicates to which

Fig. 2. Cost evaluation without optional control job (top) or with optional control job at time t (bottom).

control loop is more beneficial the optional control job allocation in terms of decreasing aggregated cost. This is formally announced in the following definitions. Definition 1 (Cost With Optional Control Job): For a given th control loop, at any given current time such that , where denotes the previous job execution time (corresponding either to a mandatory or optional job), and denotes the next mandatory job execution time, the cost that by executing would be achieved in the time interval an additional -optional control job at time [Fig. 2 (bottom)] is given by (15) Definition 2 (Cost Without Optional Control Job): Under the same conditions given in Definition 1, the cost that would be by not executing an adachieved in the time interval ditional -optional control job at time [Fig. 2 (top)] is given by (16) Definition 3 (Total Cost With One Optional Control Job): For a set of control loops, the total aggregated cost as in (14) that by executing would be achieved in the time interval in the th control loop an additional -optional control job at is given by time such that (17) and are defined as in Definition 1. where At this point, the allocation policy can be defined as follows. Definition 4 (Minimum Cost Policy—MC): Considering control loops, each one with a plant in the form of (1) whose control signal is given by (3) for both mandatory and optional control jobs, and whose control performance is evaluated via (4)–(6), provided that each optional control job can be only allocated to one control loop at a time if bandwidth is available, such that , the at any given current time th-optional control job is allocated to the th-control loop given by (18) where and are defined as in Definition 1. The approach adopted in this paper for implementing the MC function in CAN borrows the observations given in [16]. By encoding in each sensor message identifier each argument

508

IEEE TRANSACTIONS ON INDUSTRIAL INFORMATICS, VOL. 6, NO. 4, NOVEMBER 2010

of the function in (18), in bus contention, the bitwise arbitration of CAN would produce the desired result: give access to the message encoding the minimum cost. However, for each th control loop, each argument depends on local paramebut also on non-local parameters such as all ters such as . In other words, the th sensor would require to have available all other sensor readings in order to compute its argument, and this is not feasible from an implementation point of view because traffic would increase and the benefits given by this technique would be lost. In the following section, we show that the MC policy can be transformed into an equivalent policy that permits their implementation by simply coding specific quantities in the sensor CAN message identifiers. The new function preserves the same logics, i.e., it identifies the same th control loop, and each quantity encoded in each sensor message priority depends only on local parameters to the sensor.

Finally, it can be observed that by simplifying in the previous from the summand, the arguments equation the term function reduces to the definition of given of the in (17). Therefore, the left-hand side of (20) is obtained. The previous theorem states that looking for the control loop providing the minimum cost is equivalent to look for the control loop providing the maximum difference. Corollary 1 (Maximum Difference Policy—MD): It follows that the MC policy given in Definition 4 can be rephrased by changing (18) to (22) In this case, the policy is named MD policy. By rewriting each argument of the function in (22), that is, by rewriting the th cost difference (19) using the detailed notation as in

B. Maximum Difference Policy Rather than identifying the control loop providing the minimum cost, the same problem can be also solved by assessing which control loop provides the maximum difference between the total aggregated cost without optional control job and each of the aggregated costs achieved when allocating the optional control job to a particular th control loop. In this case, from the cost differences, the maximum indicates to which control loop is more beneficial the optional control job allocation. This idea is formalized next. , the cost differDefinition 5: In the time interval ence that for the th control loop would result from executing the -optional control job is defined from (15) and (16) as (19) Theorem 1: Under the assumptions taken in the MC policy (Definition 4), it holds that

(20) Proof: Starting with the right hand side term of the equality (20), and considering that

holds, the definition of given in (19) can be placed into the right-hand side of the previous equality and rearranged as (21) Observing that adding a constant to the arguments of a function does not alter its result, the expression is added to the arguments of the function in (21), which results in

it can be observed that the arguments of the function in , are much simpler that the arguments of the (22), function in (18), . However, more important, it can also can be computed at be observed that each cost difference runtime by each th closed-loop sensor node without requiring data from the other sensor nodes. This was not the case for the arguments of the function in (18) because each argument required to know, among other parameters, all the control loops’ sensor readings. Therefore, for the MD policy, if all sensor nodes encode in their optional sensor messages’ priorities the 1’s complement , in bus contention, the bitwise arbitraof their specific tion phase will allow bus access to the control loop providing the maximum difference, that is, providing the minimum aggregated cost. C. Message Encoding In the operation of each control loop, mandatory control jobs imply the periodic transmission of sensor and control messages, which must be guaranteed and must become the highest priority type of messages because they grant controlled plants stability. Then, it is desired that messages with non-control data will behave as expected in terms of being granted bus access once control messages and mandatory sensor messages have been guaranteed. Finally, sensor messages for optional control jobs, as well as the corresponding control messages, will be in position to contend for the bus after all other types of messages have been guaranteed, that is, using the spare bandwidth. The accomplishment of these functionalities is achieved by deploying each networked control system using a simplified version of the application profile presented in [22] for messages with 29-bit identifier. Specifically, as illustrated in Fig. 3, four messages classes are defined, which can be distinguished by the first three bits of the message identifier. The highest priority class, starting with 000 in the message identifier, con-

MARTÍ et al.: RUNTIME ALLOCATION OF OPTIONAL CONTROL JOBS TO A SET OF CAN-BASED NETWORKED CONTROL SYSTEMS

Fig. 3. Bit encoding for message classes.

tains control messages for both mandatory and optional control jobs. They are always granted to win bus access, they are triggered by sensor messages, and they are used by controller nodes to send control signals to actuator nodes. The following higher priority class, starting the identifier field with 001, corresponds to the mandatory sensor messages, which have to be also always granted in the bus. These messages, that are used by sensor nodes to periodically start mandatory control jobs, are triggered by the periodicity given by . The next class, with identifier starting with 010, contains the general purpose messages and they have to be also guaranteed. Finally, the lowest priority class, starting with 011 in the identifier, contains the optional sensor messages. They are not guaranteed and they are used by all sampler nodes to trigger optional control updates at whenever bandwidth is available. the rate given by Apart from the specific first 3 bits, the following bits of the message identifier will encode, in the case of control messages, the 1’s complement of the control signal, and the sampler node identifier. Hence, messages with higher control signals are allowed to win access on the bus in a predictable fashion in case of collisions with other control messages. In the case of sensor messages (either mandatory or optional), the following bits of the message identifier encode the 1’s complement of (named control value in Fig. 3), and the sensor node identifier. For mandatory sensor messages, this encoding also solves collisions between messages of this class in a predictable fashion. If the encoded control signals are exactly the same, identifiers further resolve the collision. For optional sensor messages, this encoding grants that the sensor node with highest difference will always win access to the bus in front of the other optional control jobs in bus contention, as specified by the MD policy, and identifiers are again used to further resolve the collision of . For both control and two messages encoding the same sensor messages, sensor nodes message identifiers are also used for filtering purposes in the controller and actuation nodes. The data field of sensor and control messages contains the information required for each closed-loop operation. Sensor messages carry the sampled state, and a timestamp if required. Control messages may carry a timestamp. Timestamps for sensor and control data may be required at the controller and actuator node to apply compensation techniques for network induced delays, as it will be further explained in Section VIII-B. The four traffic classes are distinguished by the first 3 bits of the message identifier. Although 2 bits would suffice (as originally presented in [22]), practical considerations prevent using only 2 bits. See [42] for details on this issue and for the type of schedulability analysis of CAN message workloads deployed

509

using this approach. The extension of the message encoding shown in Fig. 3 to CAN 2.0 A is straightforward but taking into account the limitation imposed by the 11-bit identifier. This will imply to have a low resolution for coding control signals and , as well as reducing the set of identifiers for sensor nodes. It has to be noted that the application of MD policy requires to have all optional sensor messages contending for the bus at the same moment whenever bandwidth is available. Hence, some sort of synchronization is required. A first approach, as described in [16], is to specify a node for performing a broadcast request (transmission of an specific message) that will trigger all the sensor nodes to contend for the medium. The drawback of this approach is that the messaging in increased. A second approach is to ensure clock synchronization between sensor nodes by applying a time synchronization protocol such as the IEEE 1588 Precision Time Protocol, as it was shown to be a feasible solution in CAN-based NCS [43]. A third approach is to observe that if the periodicity for optional sensor messages is shorter than the transmission time of an optional sensor message, the contention for the bus will automatically occur. The approach taken in this paper uses the third approach whenever the condition applies. Hence, the adopted synchronization technique is based on overloading and having all sensor nodes attempting to transmit before the end of any current transmission. The submitted load is then over 100% which leads to message drops with the bus load kept at 100%. As a consequence, the messages that are dropped are always the lower priority ones of optional control jobs, thus not having any impact in the mandatory control jobs or on non-control messages. When the periodicity for optional sensor messages is larger than the transmission time of an optional sensor message, some sort of precise clock synchronization must be enforced, such as hardware time-stamping support. Otherwise, priority inversions may occur and the wrong optional control job might be executed. This, however, does not have a critical impact, just a degradation of the expected control performance improvement. The impact of these priority inversions will be addressed in future work. Finally, it has to be noted that the message encoding presented in this section works when control loops operate in a low range values. However, for widely varying , it may of different be better to allocate priorities following other principles such as allocating according to rather than according to whether it is a sensor or control message. Otherwise, penalties in schedulability and most likely in control performance may occur. V. CONTROLLER DESIGN FOR OPTIONAL CONTROL JOBS Reducing the overall aggregated cost by allocating as many optional control jobs as possible can be only achieved if the design of the control gains for optional control job ensures such a property. In the following, two different strategies are presented, and their pros and cons outlined. A. Accelerable Gain Design This strategy borrows the results found in [17] where a controller gain design method based on the Bellman optimality principle for optional control jobs, originally named accelerable

510

IEEE TRANSACTIONS ON INDUSTRIAL INFORMATICS, VOL. 6, NO. 4, NOVEMBER 2010

control tasks, was presented. For any -optional control job executing at time , and taking into account that the next manda, with , the control gain that tory job will execute at applies in (3) is , which is designed using (22) and (23) in [17], and considering that the control signal should . be held from to This design strategy has the property that the more control updates are performed within mandatory control jobs, the better is the control performance. This is formally summarized using the notation of this paper in the following lemma. Lemma 1: For a given th control loop, the design of optional control jobs using (22) and (23) in [17] ensures that, in the time interval (23) denotes the time of the possible exewhere cution of a -optional control job, and where and denote the previous job execution time (corresponding either to a mandatory or optional job) and the next mandatory job execution time, respectively. Proof: If follows from [17]. Remark 1: By applying Lemma 1 to the cost difference defined in (5), it can be observed that (24) which means that the application of any optional control job is always beneficial in terms of reducing cost. By noting that the number of optional control jobs between , the application of the mandatory control jobs is presented allocation policy using this control design strategy, named MDA (Maximum Difference with Accelerable gain design) policy, requires to compute for the controller node a control gains for optional control number of jobs while only one gain is required by mandatory control jobs. B. Invariant Gain Design This strategy, rather than using specific design methods for gains of optional control jobs, specifies the same invariant control gain designed using standard optimal control methods for both mandatory and optional control jobs. Therefore, for any -optional control job executing at time , the control gain , being the same LQR that applies in (3) is gain found for mandatory control jobs. However, as outlined in [17], by using invariant control gains there is no guarantee that a single optional control job execution will improve the control performance. To overcome this limitation, each sensor node, apart from , checks its sign. If computing the cost difference , that is, if executing the optional control job guarantees control performance improvement, then the optional sensor message is placed for transmission. Otherwise, the optional sensor message is discarded. Between all the optional sensor messages placed for transmission, the MD policy will work as specified in (22): it will give bus access to that message whose cost difference is maximum.

In summary, the application of the presented allocation policy using the invariant gain design strategy, named MDI (Maximum Difference with Invariant gain design) policy, requires to compute only one control gain, which requires less memory at the controller node, at the expenses of possibly reducing the achievable aggregated control performance improvement because for each control loop a fewer number of optional control jobs will be able to content for bus access. VI. COMPUTATIONS AT SENSOR NODES The implementation of the presented allocation policy, for both optional control gain design strategies, MDA and MDI, (19) at each sensor requires computing the cost difference node. This section describes alternative methods for computing and outlines their resource requirements, both in terms of computational and memory demands, which basically reduce to the number of matrix operations (either matrix-by-vector or vector-by-vector multiplications) and the number of matrices to be stored in memory. A. Preliminaries The cost difference in (19) is the difference between subcosts in (16) and in (15), where each one has a term which is an infinite summand of the form as defined in (13). This infinite summand reflects , only the cost achieved by executing, from a given time mandatory control jobs. Noting that the gain for mandatory control jobs for both MDA and MDI are designed using standard LQR control theory, and noting also that mandatory control jobs execute periodically with a sampling interval given by , it is well known from standard optimal control theory for discrete-time systems [41] that the following equation holds: (25) is the solution to the algebraic Riccati equation, and in (25) varies depending on the previous job execution time with respect to the first mandatory job executing at time . Hence, can be calculated as where

(26) where (27) Hence, terms

can be computed as in (25).

B. Direct Method as indicated in (19) This method consists on evaluating and considering the simplification (25). That is, for both the MDA and MDI policies, at each sensor node, the cost difference is computed from sub-costs (15) and (16) where their infinite summand can be calculated as in (25) with (26)–(27) and the rest of summands of the can be calculated using (6). form

MARTÍ et al.: RUNTIME ALLOCATION OF OPTIONAL CONTROL JOBS TO A SET OF CAN-BASED NETWORKED CONTROL SYSTEMS

In terms of computations, for each , six matrix opera, three matrix operations are required. For each and matrix tions are required. Hence, operations are required for and , respecmatrix operations tively. Summing up, a total of . The only difference between are required for computing MDA and MDI is that in MDI the controller gain that applies in the computations is always the same. Looking at the number of matrices to be stored in memory, the , the three counting is the following. For computing any , and have to be precomputed matrices for all possible between two consecutive mandatory control jobs. By remembering that the number of possible optional control jobs between two consecutive mandatory control jobs is , a total of matrices needs to be precomputed. For computing any must be precomputed, and in addition, a number of mahave to be precomputed due to the same logics trices as before. Summing up, a total of matrices have to be precomputed. In summary, for applying both policies using this method, namely, MDAd and MDId, at each sensor node, and for each optional control job, a total of 24 matrix operations have to be performed while requiring to have stored in sensor memory a matrices. By applying simple number of , algebraic manipulations to the defining expressions of the number of matrix operations can be reduced to 15. C. Improved Method

and

(30) Proof: By noting that the term (16) can be rewritten as follows:

(28)

for computing

(31) the cost difference (19) directly transforms to (32) (33) (34) and (34) In the previous equation, (33) corresponds to . corresponds to Focusing in (33), the term can be also as . In written using the known value addition, by using (25), its right summand is defined as (35) can be computed based only on

where

in (19) drasUsing this method, the computation of tically simplifies the required number of matrix operations and the number of matrices to be stored in sensor memory. The main , that is, to compute subcosts difference is to compute in (16) and in (15), over rather than . This can be done because the plant evolution over the is identical when computing time interval and . In addition, each sensor node can memorize the (regardless of whether last updated control input, namely it was updated by a mandatory or an optional control job) from can be used instead the previous control message. Hence, of requiring keeping track of in (25). This is formalized as follows. being the value of the previous Proposition 1: With control update for a given optional instant , the maximum difin (19) can be alternatively computed as ference

511

and

as

(36) By considering (35) and (36), and by noting that using the compact notation (10) the left summand of (33) is defined as (37) expression (33) can be computed as (38) is given in (30). where Focusing in (34), its left summand using the compact notation (10) is defined as

where

(39) (29)

which can be rewritten using

as

with (40)

512

IEEE TRANSACTIONS ON INDUSTRIAL INFORMATICS, VOL. 6, NO. 4, NOVEMBER 2010

Considering (25) and (26) with (34) is

, the right summand of

jobs designed using the accelerable gain design strategy are, respectively

(41) where

can be computed as

(42) Hence, using (40), (41), and (42), expression (34) can be computed as (43) or alternatively as

For successive control job executions, the previous gains apply whenever the job is a mandatory job or one of the two posis sible options, e.g., the gain for the mandatory job at , and the gain for the following optional job, , is . For both control loops, consider the following scenario with : current time

(44) where is given in (30). Substituting (38) and (44) into (32), the definition of given in (28) and (29) is obtained. That is, can be effectively computed only considering . the time interval By simple inspection of (28), the required number of matrix operations at each sensor node for each optional control job reduces to 3, and the number of matrices to be stored in memory , which correspond to the number of reduces to to be precomputed for all than can occur matrices between two consecutive mandatory control jobs. In summary, the MDA and MDI policies using this method, that will be denoted as MDAi and MDIi, drastically reduce the resource demands compared to the direct method. VII. NUMERICAL EXAMPLE This section illustrates using a simple NCS example some of the concepts introduced in the preceding sections. , each Consider two equal networked control loops, one controlling a plant in the form of

where , and , respectively, represent mandatory control job executions, non-executed optional control job, and optional control job to be executed. In other words, for both control loops, the first mandatory control job executed at (with ), the first optional control job at time did not execute, and at the current time , the second op) knowing that the tional control job can execute (with next mandatory control job will execute at . According to the MD policy (Section IV-B) with accelerable gain design (Section V-A, MDA), the application of the presented approach . would require solving By knowing that the current sampled states and last updated control signals at each sensor node are , and , , respectively, each is computed using the improved method difference (Section VI-C, MDAi) as

(45) where and but with different initial conditions, . For both controllers, the cost function is characterized by (46) For simplicity, assume that both execute mandatory control jobs with and possible optional control jobs with . With these settings, and as indicated in Section V, for both controllers, the control gain for the mandatory jobs designed using LQR design and the gains for possible optional

Hence, the MDAi policy will grant bus access to the 2-loop because its optional control job execution will provide the maxthat is used imum cost difference. Apart from matrix , an additional matrix for those optional jobs at times has to be stored in each sensor memory (2 matrices in total) for performing the same computations for the possible ex. ecution of optional jobs at times

MARTÍ et al.: RUNTIME ALLOCATION OF OPTIONAL CONTROL JOBS TO A SET OF CAN-BASED NETWORKED CONTROL SYSTEMS

513

Fig. 5. Electronic double integrator circuit.

Fig. 4. NCS experimental setup. (a) Networked tower. (b) A sensor/actuator board.

Note that this result coincides with the approach using the MC policy (Section IV-A). In this case

shows one single sensor/actuator board. It can be observed that it has also plugged a small board that contains a double integrator electronic circuit, as well as, CAN transceivers. Each plant is an electronic double integrator, as illustrated in k Fig. 5. With component values nF, and k , each electronic circuit is modeled as in

(47)

Hence, the execution of the optional control job for the two-loop provides the minimum cost. VIII. PROOF-OF-CONCEPT IMPLEMENTATION AND EXPERIMENTAL RESULTS As a prototype and performance demonstration, a proof-ofconcept implementation consisting on the CAN-based networked control of three plants, each of the form of a double integrator electronic circuit, has been built and extensive experiments have been performed. Although only few specific cases are summarized, the experiments are general and the results are not limited to these or any other special cases. A. Experimental Setup The experimental setup is shown in Fig. 4(a). It consists on a tower of seven boards. Each board is a Flex board [44] equipped with a dsPIC microcontroller running the Erika [44] real-time kernel. Specifically, the microcontroller is a dsPIC33FJ256MC710 equipped with two CAN ports [45]. In Fig. 4(a), the three boards at the bottom are the controllers, and the following three boards are the sensors/actuators. All of them are connected with a single CAN bus. Although that for simplicity each pair of sensor/actuator share the same physical board, any control loop operation requires using the CAN network as specified, that is, it requires sending the sample from a sensor/actuator board to a controller board, and sending the control signal from the controller board back to the sensor/actuator node. The board in the top is used for debugging purposes and for generating non-control messaging when required. It has plugged two small boards used also for debugging. Fig. 4(b)

. where the state vector is The goal of the controller is to make the circuit output voltage ( in Fig. 5) to track a reference signal by giving the approand priated voltage levels (control signals) . Both states are read via the Analog-to-Digital-Converter (ADC) port of the microcontroller and is applied to the plant through the Pulse-Width-Modulation (PWM) port. Table I details the main software tasks in each board. It has to be pointed out that other tasks than those listed in Table I have been also coded in each board for debugging purposes. The task period for the Send 001 RT TASK is given by the period of mandatory control jobs, . The task period for the Try to Send 011 background activity is given by the period of optional control jobs, . The Read State is not periodic but activated whenever sampling a plant is required, that is, during the execution of the previous two tasks Send 001 and Try to Send 011. The period for the Reference Change RT TASK is of several orders of magnitude higher than . However, it is adjustable to permit random set-point changes start times. The Compute Control Send 000 is activated by the reception of an upcoming sensor message. The two tasks Receive Control and Apply Control are further explained in next subsection. The accomplishment of the functionality mandated by the MD allocation policy through the message encoding presented in Section IV-C requires controlling the automatic retransmission functionality of CAN. For optional sensor messages, the automatic retransmission is disabled to avoid retransmissions upon arbitration loss. For mandatory transmissions, the automatic retransmission is on in order to enforce the transmission of these messages in case of collisions. Each circuit is constantly affected by perturbations in terms of step set-point changes. They occur at random times (within

514

IEEE TRANSACTIONS ON INDUSTRIAL INFORMATICS, VOL. 6, NO. 4, NOVEMBER 2010

TABLE I SOFTWARE TASKS IN EACH BOARD. LOAD, IN THE TOP BOARD, IS ONLY ACTIVATED WHEN REQUIRED

a given interval) in order to capture all possible different scenarios. Two main scenarios are reported: non-overlapped and overlapped. In the first one, set-point changes affect each plant at different times in such a way that in general only one plant is in transient at a time. In the second one, changes affect the three plants at the same time. This two scenarios represent the extreme cases and therefore, any other evaluated scenario provides performance numbers lying within the presented ones. B. Control Design Considerations A network within a control loop introduces sampling-to-actuation (s/a) delays [9]. In the theoretical approach delays were omitted to focus only in the allocation problem. However, varying s/a delays have to be considered, otherwise, performance degradation may occur. Considering the controlled plants dynamics, for high bus baudrates, delays have been shown to be negligible. However, for low bus baudrates, delays affect performance. To solve this problem with a single technique, the approach adopted in this paper is to apply control updates at the actuator node at synchronized instants provided that synchronization between networked nodes is implemented. Synchronized actuation instants requires activating the Apply Control task in the actuator node at a constant time with respect to the corresponding sampling time in such a way that s/a delays can be treated as being constant. To do so, for each bus baudrate, an upper bound to the s/a delay that appear in the implementation can be inferred from direct measurements and taking into account messages worst case response times extracted from the schedulability analysis [42], namely, . Then, for both mandatory and optional control jobs, the sampling time at the sensor node is recorded, namely , and sent in the sensor message. The controller, after computing the control signal, sends it to the actuator node together with the sampling time. Upon reception at the actuator node by the task Receive Control, the current time is recorded, namely, , and the Apply Control task is programmed to fire after time units. Hence, the control signal is applied to the plant at time , thus forcing a constant s/a delay. Note that giving a correct value to represents a tradeoff between performance and synchronization issues. It must be short to minimize the degradation that it introduces, while it has to be long enough to ensure that control messages will always arrive before the time at which the control action must be applied. Oth-

erwise, an old control action will be applied (vacant sampling scenario). Since for any bus rate from 50 to 1000 Kb/s, the s/a delays are shorter than the sampling periods considered in the implementation, the standard extended model including a time delay with [41] can be used for controller design purposes. Looking at the presented approach, this would imply including an input time delay in the plant model (1), which would propagate to the cost equations with an additional dimension. In general, if an implementation produces delays longer that the sampling period, the extended model including a time delay with [41] can also be used. The adopted strategy for dealing with varying time delays imposes a constant s/a delays within each control loop. As a consequence, an small but predictable control performance degradation still occurs. An alternative strategy that even removes this degradation could be to apply the one shot task model [46] in a networked setting [47]. A complementary and extended analysis of the tradeoffs between periods, delays and performance in NCS can be found in [48]. C. Standard Control Design Case Although different networked control scenarios are evaluated in terms of different control parameters and bus conditions, this section details the main characteristics for the standard case. For this case, the bus rate is 50 Kb/s, the sampling period for mandatory control jobs for the three loops is the same and equal to 50 ms, and the sampling period for optional control jobs is always 1 ms. From the measured time delay 5.2 ms, its upper bound has been defined as ms. For the controller design, the cost function weighting matrices are the identity. Set-point changes periodicity is in average set to 1.5 s, and the cost evaluation in sensor nodes is performed using the MDAi method. Many of the performance results presented next are reported in percentage improvement with respect to the static case. The static case corresponds to the traditional approach to the analysis and design of NCS where only mandatory control jobs are executed, i.e., pure periodic networked control. With these settings, the required bandwidth for the static case is about 25%, and the rest is left for executing optional control jobs using the MDAi. It has to be noted that in the majority of the experiments reported next, the objective is to characterize the presented allocation policy, and in particular, to assess the influence of key

MARTÍ et al.: RUNTIME ALLOCATION OF OPTIONAL CONTROL JOBS TO A SET OF CAN-BASED NETWORKED CONTROL SYSTEMS

515

Fig. 7. MDA performance with respect to the static policy.

where the integral is computed using trapezoidal numerical integration, and is the simulation time. D. MDA Basic Performance Analysis

Fig. 6. Type of dynamics achieved with only mandatory control jobs (Static) or with MDA policy. (a) Static dynamics. (b) MDA dynamics.

parameters on the achievable control performance. Many of the results are presented in percentage improvement with respect to the static policy. The reason for using this metric is not for comparing our approach to the static approach. It would not be fair because the static only uses 25% of the bandwidth while MDAi uses up to 100%. The reason is only for having an starting performance reference value from which to assess the type of improvements that can be achieved by MDAi under different conditions. In addition, it has to be also noted that the percentage of available bandwidth for optional messages is kept constant in average in order to facilitate assessing again performance improvements. However, the allocation approach, as outlined in the introduction, does not require to know a priori this percentage for its correct operation. In all the cases, performance evaluation in terms of aggregated cost is performed by adding for each networked control loop the continuous counterpart of the discrete cost function used for controller design

In particular, each was calculated taking values for and every 10 ms, and accumulating the integral over intervals of 10 ms as in

For illustrative purposes, Fig. 6 shows the type of dynamics that are obtained using the static approach [Fig. 6(a)] and the MDAi approach [Fig. 6(b)]. It can be appreciated that in the static approach, control inputs change every 50 ms, while in the MDAi approach, they change quicker due to the application of optional control jobs. In addition, the MDAi achieves faster dynamics in both state and control signals. Fig. 7 shows accumulated cost for the three plants over time for the MDAi (labeled “dynamic”) and static policies. The MDAi provides a lower curve, that is, it is able to effectively use the available bandwidth to improve control performance with respect to the static case. The MDAi performance shown in the green curve of this figure is for the case of non-overlapped perturbations. A third blue curve that corresponds to the MDAi case when perturbations are overlapped completes this analysis. Although the performance decreases, it is still below the static case. The first conclusion that can be drawn is that the presented approach is capable of improving control performance with respect to the static (standard) approach if bandwidth is available, as in the case presented here. From Fig. 7 (green curve), it can be seen that up to 20% of performance improvement is achieved when using the available portion of 75% of the spare bandwidth. E. MDA Performance for Different Control Settings Fig. 8(a) shows the relative performance improvement of the MDAi with respect to the static approach for different sampling periods for the mandatory jobs of the three control loops. For each sampling period, the delay upper bound is specified according to the value that applies. As in the previous case, the CAN network is only used for sending control data. Hence, for each run, the given period for mandatory jobs determines the portion of available bandwidth left in average for optional control jobs. The figure shows that as the sampling period for mandatory jobs increases, the performance improvement also increases, for either overlapped and non-overlapped

516

IEEE TRANSACTIONS ON INDUSTRIAL INFORMATICS, VOL. 6, NO. 4, NOVEMBER 2010

Fig. 9. MDA performance for different bus baudrates.

dedicated to mandatory control jobs, and therefore, to decrease the available bandwidth dedicated to other non-control messaging. If many applications require using the bus, it may be more appropriated to assign long sampling periods to mandatory control jobs for achieving acceptable minimum control performance, and then, after guaranteeing the other applications messages, use any available bandwidth for placing in the bus the optional control jobs. Fig. 8(c) shows the relative performance improvement of the MDAi with respect to the static approach for different values , which is the weighting matrix associated to the control of signal in the cost function, when the sampling period for mandams. As it can tory control jobs has the standard value, be seen in the figure, the performance improvement decreases as increases. Increasing implies forcing less aggressive control signals, which results in slower plant dynamics. However, more important, increasing implies that the difference between mandatory control gains and optional control gains becomes smaller. Therefore, the application of an optional control job gives a low benefit compared to the performance already achieved by a mandatory control job. F. MDA Performance for Different Bus Conditions

Fig. 8. MDA performance for different control settings. (a) Different sampling periods (relative). (b) Different sampling periods (absolute). (c) Different values.

Q

perturbations. This is an expected result in the sense that in general terms, the lower the execution rate for mandatory jobs, the higher the cost for the static approach. Hence, adding optional control jobs, in percentage, results more and more beneficial. Fig. 8(b) gives an alternative view of the performance study for different periods by showing the absolute performance values of the aggregated cost for the three plants. The first conclusion that can be drawn is that the best approach is to give short sampling periods for mandatory control jobs. However, this implies to increase the portion of guaranteed bandwidth

Fig. 9 shows the relative performance improvement of the MDAi with respect to the static approach for different CAN bus rates for the standard case (Section VIII-C). As it can be seen in the figure, as the bus rate increases, the relative performance improvement also increases. The main reason for this behavior relies in the fact that the higher the bus baudrate, the faster is the control messaging, and therefore, more room is created for sending optional control jobs and improving control performance. It is also worth noting that for high bus baudrates, between a sensor message and the corresponding controller message, other sensor messages access the bus because the computation time of the controller is longer than the CAN interframe time. The only negative effect of this situation is that the s/a delays increase. But if they are appropriately upper bounded and handled as explained in Section VIII-B, the performance improvement offered by MDAi positively compensates the known degradation caused by these delays. In all the previous performance figures, the whole CAN bus was dedicated to control messaging. Fig. 10(a) analyzes

MARTÍ et al.: RUNTIME ALLOCATION OF OPTIONAL CONTROL JOBS TO A SET OF CAN-BASED NETWORKED CONTROL SYSTEMS

517

Fig. 11. Static versus MDAi when both occupy 100% of the bandwidth.

Fig. 10. MDA performance versus bandwidth. (a) Bandwidth dedicated to control. (b) Different thresholds.

for the standard case under the scenario of non-overlapped perturbations, the performance improvement depending on the percentage of bandwidth dedicated to control messaging. For this experiment, the Load task that injects periodic non-control messages in the bus is active and it is given an adjustable period in order to permit regulating the bandwidth left for control messages. First of all, it has to be noted that at 50 Kb/s, the bandwidth required to only execute mandatory control jobs is about 25%. From this point, as the bandwidth dedicated to optional control jobs increases, the achievable performance improvement increases. However, more important, having only a small amount of extra bandwidth quickly implies a fast performance increase, as it can be seen in the interval going from 25% to 40%. In this experiment, for each run, the percentage of available bandwidth for optional control jobs is kept constant in average to facilitate analyzing performance improvement as a function of this percentage. However, fixing a percentage of spare bandwidth dedicated to optional control jobs does not mean that it is known a priori when the bandwidth will be available for them. It appears randomly, and optional control jobs are accommodated in the bus as soon as the bus is idle (and not required for higher priority messages. An alternative view of the previous result is shown in Fig. 10(b). In this experiment, using the standard case under the MDAi policy, optional sensor messages triggering optional control jobs are only trying to access the bus when the

is above a given threshold. The maximum difference immediate consequence is that each networked control loop tries to send less optional control jobs and, therefore, the bus load due to optional control jobs is drastically decreased while performance improvement is still achieved. As expected, as the threshold increases, the performance improvement decreases but the occupied bandwidth also decreases. The main conclusion drawn from Fig. 10(a) and (b) is that control performance improvement can still be achieved without requiring a high amount of bandwidth availability. G. MDA Versus Static Using Both 100% of the Bus Bandwidth In the previous performance results, many of the control performance improvements achieved by the MDAi were assessed compared to the case where only mandatory jobs are executed, that is, the static policy. In particular, for those experiments ms, the MDAi where mandatory control jobs had used up to 100% of the bandwidth while the static only 25%. Although it is not the objective of the allocation approach presented in this paper to outperform the static policy when both use 100% of the bandwidth (or the same portion of bandwidth), studying this scenario brings interesting results that further characterize the presented policy. To study this case, the following experiment has been performed, which is illustrated in Fig. 11. On one hand, the static approach has been configured in such a way that the three ms, which represent a controllers execute with 100% of processor utilization. On the other hand, the MDAi executes mandatory jobs as in the standard case, that is, with ms, occupying 25% of the bandwidth, while the other 75% of the bandwidth is allocated to optional control jobs, and this setup has been tested for two perturbation scenarios: i) the non-overlapped scenario (already used in many experiments and labelled here as ) and ii) the scenario where only one plant is affected by all the previous perturbations that appeared in the scenario while the other two remain in the equilibrium (labeled here as ).

518

IEEE TRANSACTIONS ON INDUSTRIAL INFORMATICS, VOL. 6, NO. 4, NOVEMBER 2010

Finally, both policies are compared to the LEF policy [22], [30] that encodes in optional sensor messages, rather than the maximum difference, the error, which is understood as a quantity reflecting the deviation of the plant state with respect to the set-point. In particular, for this experiment, optional sensor messages encode . The performance results indicate that the approaches presented in this paper perform better than the LEF. It has to be outlined that this comparative performance analysis does not explicitly includes the MEF-TOD policy [29] because in terms of performance, it can be considered similar to LEF. IX. CONCLUSION Fig. 12. Comparative performance analysis.

Fig. 11 shows that for the non-overlapped scenario ( ), the static outperforms the MDAi. This is an expected result because when all plants are continuously affected by perturbations, assigning bandwidth at runtime using a non-optimal policy, in average, cannot offer significant benefits. Note that for a static portion of available bandwidth, the static policy is optimal while the MDAi has not been designed to be optimal. Moreover, it has not been designed considering that the portion of available bandwidth is fixed and kept constant. However, it is worth noting that the difference in terms of control performance between them is scenario the MDAi not big. Also, interesting is that for the performs better than the static (not plotted in the figure because its performance is exactly the same achieved by the static ), corroborating the existing trend already reported in the literature and stressed in Section II which indicates that runtime allocation of resources as a function of the controlled plants’ dynamics is the best approach when several controllers execute using limited computing resources. Finally, it should be noted that the smaller the portion of bandwidth assigned to both policies, the slower will be the rate for messages in the static policy. However, for the MDAi, if guaranteed messages still have a slower rate than the applied in the static, optional control messages can still be temporarily transmitted with a fast rate (very fast compared to the static one), thus providing more benefit with respect to the static policy. H. MDA, MDI, and Comparison to Other Approaches All the previous performance figures were obtained by running the three networked control loops with the MDAi policy. That is, control gains for optional control jobs were designed as accelerable gains, as described in Section V. However, the same section offered another alternative control design strategy, MDI, based on using invariant gains. Although not shown here, the previous tendencies and performance numbers are similarly obtained by the MDI policy, with a small difference: the amount of performance improvement is lower than for the MDA policy. To illustrate this difference, Fig. 12 shows for the standard case the performance numbers for the MDAi and MDIi. As announced, the MDA performs better than the MDI in both perturbation scenarios. The only benefit of the MDI is that it facilitates the implementation in the sense that only one control gain has to be stored in each controller node.

This paper has presented an approach to CAN-based NCS that effectively permits to increase overall control performance by adequately placing in the bus additional control jobs between the guaranteed standard periodic messaging whenever bandwidth is available. The theory underlying the approach establishes the rule for placing the additional control messaging, and the CAN infrastructure permits to implement the rule with a bounded time complexity. An analysis of the computation and memory demand of the presented approach offers different alternative computational methods for a feasible and efficient implementation. The approach has been evaluated using a proof-of-concept implementation and the extensive experiments show that: i) the presented approach is able to improve control performance with respect to the case where only periodic messaging is used, thus corroborating the theoretical approach; ii) the approach can provide increased control performance as soon as a little bit of spare bandwidth is available; and iii) the comparative analysis with previous work indicates that the presented approach performs the best. Future work will cover several topics not addressed in this paper. First of all, the effect of noise could also be accounted for in the controller design as well as in the cost evaluation. Second, the computational demand of observers could be included if all plant states were not available. Third, if the sensor and/or controller node information to be transmitted would not fit in a single CAN message, the message codification should be updated accordingly. Finally, the formulation of the same solution in an event-based setting is also pursued. REFERENCES [1] CAN Specification Version 2.0. Robert Bosch GmbH. Stuttgart, Germany, 1991, Postfach 30 02 40, D-70442. [2] M. Bertoluzzo, G. Buja, and R. Menis, “Control schemes for steer-bywire systems,” IEEE Ind. Electron. Mag., vol. 1, no. 1, pp. 20–27, Spring 2007. [3] F. Baronti, F. Lenzi, R. Roncella, R. Saletti, and O. Di Tanna, “Electronic control of a motorcycle suspension for preload self-adjustment,” IEEE Trans. Ind. Electron., vol. 55, no. 7, pp. 2832–2837, Jul. 2008. [4] H. A. Hansson, T. Nolte, C. Norstrom, and S. Punnekkat, “Integrating reliability and timing analysis of CAN-based systems,” IEEE Trans. Ind. Electron., vol. 49, no. 6, pp. 1240–1250, Dec. 2002. [5] G. Cena and A. Valenzano, “A protocol for automatic node discovery in CAN open networks,” IEEE Trans. Ind. Electron., vol. 50, no. 3, pp. 419–430, Jun. 2003. [6] J. Ferreira, L. Almeida, A. Fonseca, P. Pedreiras, E. Martins, G. Rodríguez-Navas, J. Rigo, and J. Proenza, “Combining operational flexibility and dependability in FTT-CAN,” IEEE Trans. Ind. Informat., vol. 2, no. 2, pp. 95–102, May 2006.

MARTÍ et al.: RUNTIME ALLOCATION OF OPTIONAL CONTROL JOBS TO A SET OF CAN-BASED NETWORKED CONTROL SYSTEMS

[7] G. Buja, J. R. Pimentel, and A. Zuccollo, “Overcoming babbling-idiot failures in CAN networks: A simple and effective bus guardian solution for the FlexCAN architecture,” IEEE Trans. Ind. Informat., vol. 3, no. 3, pp. 225–233, Aug. 2007. [8] G. Rodriguez-Navas, S. Roca, and J. Proenza, “Orthogonal, fault-tolerant, and high-precision clock synchronization for the controller area network,” IEEE Trans. Ind. Informat., vol. 4, no. 2, pp. 92–101, May 2008. [9] Y. Tipsuwan and M. Y. Chow, “Control methodologies in networked control systems,” Control Engineering Practice, vol. 11, pp. 1099–1111, 2003. [10] K. C. Lee, S. Lee, and M. H. Lee, “QoS-based remote control of networked control systems via Profibus token passing protocol,” IEEE Trans. Ind. Informat., vol. 1, no. 3, pp. 183–191, Aug. 2005. [11] M. E. M. Ben Gaid, A. Cela, and Y. Hamam, “Optimal integrated control and scheduling of networked control systems with communication constraints: Application to a car suspension system,” IEEE Trans. Control Syst. Technol., vol. 14, no. 4, pp. 776–787, Jul. 2006. [12] S. K. Mazumder, M. Tahir, and K. Acharya, “MasterSlave currentsharing control of a parallel DCDC converter system over an RF communication interface,” IEEE Trans. Ind. Electron., vol. 55, no. 1, pp. 59–66, Jan. 2008. [13] K. Natori, R. Oboe, and K. Ohnishi, “Stability analysis and practical design procedure of time delayed control systems with communication disturbance observer,” IEEE Trans. Ind. Informat., vol. 4, no. 3, pp. 185–197, Aug. 2008. [14] H. Li, M.-Y. Chow, and Z. Sun, “EDA-based speed control of a networked DC motor system with time delays and packet losses,” IEEE Trans. Ind. Electron., vol. 56, no. 5, pp. 1727–1735, May 2009. [15] D. Ne˘sic´ and D. Liberzon, “A unified framework for design and analysis of networked and quantized control systems,” IEEE Trans. Autom. Control, vol. 54, no. 4, pp. 732–747, Apr. 2009. [16] B. Andersson, N. Pereira, W. Elmenreich, E. Tovar, F. Pacheco, and N. Cruz, “A scalable and efficient approach for obtaining measurements in CAN-based control systems,” IEEE Trans. Ind. Informat., vol. 4, no. 2, pp. 80–91, May 2008. [17] M. Ben Gaid, D. Simon, and O. Sename, “A design methodology for weakly-hard real-time control,” in Proc. 17th IFAC World Congress on Automatic Control, Seoul, Korea, Jul. 2008. [18] D. Hristu-Varsakelis and W. S. Levine, Handbook of Networked and Embedded Control Systems. Boston, MA: Birkhäuser, Jun. 2008. [19] H. Rehbinder and M. Sanfridson, “Scheduling of a limited communication channel for optimal control,” Automatica, vol. 40, no. 3, pp. 491–500, Mar. 2004. [20] P. Martí, C. Lin, S. A. Brandt, M. Velasco, and J. M. Fuertes, “Draco: Efficient resource management for resource-constrained control tasks,” IEEE Trans. Comput., vol. 58, no. 1, pp. 90–105, Jan. 2009. [21] M.-M. Ben Gaid, A. Çela, and Y. Hamam, “Optimal real-time scheduling of control tasks with state feedback resource allocation,” IEEE Trans. Control Syst. Technol., vol. 17, no. 2, pp. 309–326, Mar. 2009. [22] M. Velasco, P. Martí, R. Castañé, J. Guardia, and J. M. Fuertes, “A CAN application profile for control optimization in networked embedded systems,” in Proc. 32th Annu. Conf. IEEE Ind. Electron. Soc., Nov. 2006, pp. 4638–4643. [23] D. Hristu-Varsakelis and P. R. Kumar, “Interrupt-based feedback control over a shared communication medium,” in Proc. 41st IEEE Conf. Decision and Control, Dec. 2002, vol. 3, pp. 3223–3228. [24] X. Wang and M. Lemmon, “Decentralized event-triggering broadcast over networked systems,” Hybrid Systems: Computation and Control HYPERLINK, vol. 4981/2008, Lecture Notes in Computer Science, pp. 674–677, 2008, DOI: 10.1007/978-3-540-78929-1_62 . [25] A. Anta and P. Tabuada, “On the benefits of relaxing the periodicity assumption for networked control systems over CAN,” in Proc. Real Time Syst. Symp., Dec. 2009, pp. 3–12. [26] A. Cervin, M. Velasco, P. Martí, and A. Camacho, “Optimal on-line sampling period assignment: Theory and experiments,” IEEE Trans. Control Syst. Technol., Jun. 2010, DOI 10.1109/TCST.2010.2053205, accepted for publication. [27] P. Pedreiras and L. Almeida, “The flexible time-triggered (FTT) paradigm: An approach to QoS management in distributed real-time systems,” in Proc. Int. Parallel and Distrib. Process. Symp., Apr. 2003. [28] J. Ferreira, L. Almeida, A. Fonseca, P. Pedreiras, E. Martins, G. Rodríguez-Navas, J. Rigo, and J. Proenza, “Combining operational flexibility and dependability in FTT-CAN,” IEEE Trans. Ind. Informat., vol. 2, no. 2, pp. 95–102, May 2006. [29] G. C. Walsh and Y. Hong, “Scheduling of networked control systems,” IEEE Control Syst. Mag., vol. 21, no. 1, pp. 57–65, Feb. 2001.

519

[30] J. Yépez, P. Martí, and J. M. Fuertes, “Control loop scheduling paradigm in distributed control systems,” in Proc. 29th Annu. Conf. IEEE Ind. Electron. Soc., Roanoke, VA, Nov. 2003, vol. 2, pp. 1441–1446. [31] D. B. Da˘cic´ and D. Ne˘sic´, “Quadratic stabilization of linear networked control systems via simultaneous protocol and controller design,” Automatica, vol. 43, no. 7, pp. 1145–1155, Jul. 2007. [32] J. Salt and P. Albertos, “Model-based multirate controllers design,” IEEE Trans. Control Syst. Technol., vol. 13, no. 6, pp. 988–997, Nov. 2005. [33] L. Yang and S. H. Yang, “Multirate control in internet-based control systems,” IEEE Trans. Syst., Man, Cybern., Part C, Appl. Rev., vol. 37, no. 2, pp. 185–192, 2007. [34] M. Santos, L. Almeida, and J. M. Fuertes, “Design space of a dual-rate switching controller,” in Proc. 7th Portuguese Conf. Autom. Control, CONTROLO 2006, Lisbon, Portugal, Sep. 2006. [35] P. Ramanathan, “Overload management in real-time control application using (m, k)-firm guarantee,” IEEE Trans. Parallel and Distrib. Syst., vol. 10, no. 6, pp. 549–559, Jun. 1999. [36] K. M. Zuberi and K. G. Shin, “Design and implementation of efficient message scheduling for controller area network,” IEEE Trans. Comput., vol. 49, no. 2, pp. 182–188, Feb. 2000. [37] M. Di Natale, “Scheduling the CAN bus with earliest deadline techniques,” in Proc. 21st IEEE Real-Time Syst. Symp., Dec. 2000, pp. 259–268. [38] G. Juanole and G. Mouney, “Networked control systems: Definition and analysis of a hybrid priority scheme for the message scheduling,” in Proc. 13th IEEE Int. Conf. Embedded Real-Time Comput. Syst. Appl., 2007, pp. 267–274. [39] T. Bai and Z. Wu, “Hybrid bandwidth scheduling for CAN-based networked control systems,” Acta Automatica Sinica, vol. 33, no. 9, pp. 963–967, Sep. 2007. [40] W. Zheng, Q. Zhu, M. Di Natale, and A. S. Vincentelli, “Definition of task allocation and priority assignment in hard real-time distributed systems,” in Proc. 28th IEEE Int. Real-Time Syst. Symp., Dec. 2007. [41] K. J. Åström and B. Wittenmark, Computer-Controlled Systems, 3rd ed. Englewood Cliffs, NJ: Prentice-Hall, 1997. [42] M. Velasco, P. Martí, J. Yépez, R. Villà, and J. M. Fuertes, “Schedulability analysis for CAN-based networked control systems with dynamic bandwidth management,” in Proc. 14th IEEE Int. Conf. Emerging Technol. Factory Autom., Mallorca, Spain, Sep. 2009, pp. 1–8. [43] P. Martí, M. Velasco, C. Lozoya, and J. M. Fuertes, “Clock synchronization for networked control systems using low-cost microcontrollers,” Automatic Control Department, Technical University of Catalonia, Res. Rep. ESAII-RR-08-2, Apr. 2008. [44] Evidence S.r.l., Flex Full Base Board, Erika Real-Time Kernel, online accessed Oct. 1, 2009. [Online]. Available: http://www.evidence.eu.com [45] Microchip dsPIC33FJ256MC710, online accessed Oct. 1, 2009. [Online]. Available: http://www.microchip.com/wwwproducts/Devices.aspx?dDocName=en024663 [46] C. Lozoya, M. Velasco, and P. Martí, “The one-shot task model for robust real-time embedded control systems,” IEEE Trans. Ind. Informat., vol. 4, no. 3, pp. 164–174, Aug. 2008. [47] C. Lozoya, P. Martí, M. Velasco, and J. M. Fuertes, “Analysis and design of networked control loops with synchronization at the actuation instants,” in Proc. 34th Annu. Conf. IEEE Ind. Electron. Soc., Nov. 2008, pp. 2899–2904. [48] W. P. M. H. Heemels, A. R. Teel, N. van de Wouw, and D. Neic, “Networked control systems with communication constraints: Tradeoffs between transmission intervals, delays and performance,” IEEE Trans. Autom. Control, vol. 55, no. 8, pp. 1781–1796, 2010, DOI 10.1109/ TAC.2010.2042352. Pau Martí (M’02) received the degree in computer science and the Ph.D. degree in automatic control from the Technical University of Catalonia, Barcelona, Spain, in 1996 and 2002, respectively. Since 1996, he has been an Assistant Professor at the Department of Automatic Control, Technical University of Catalonia. During his Ph.D., he was a visiting student at Mälardalen University, Västerås, Sweden. From 2003 to 2004, he held a research fellow appointment at the Computer Science Department, University of California at Santa Cruz. His research interests include embedded systems, control systems, real-time systems, and communication systems.

520

IEEE TRANSACTIONS ON INDUSTRIAL INFORMATICS, VOL. 6, NO. 4, NOVEMBER 2010

Antonio Camacho received the B.S. degree in chemical engineering and the M.S. degree in automation and industrial electronics from the Technical University of Catalonia, Barcelona, Spain, in 2000 and 2009, respectively. Currently, he is working towards the Ph.D. degree in electronic engineering at the Technical University of Catalonia. His research interests include networked and embedded control systems, industrial informatics, and power electronics.

Manel Velasco (M’08) graduated in maritime engineering in 1999 and received the Ph.D. degree in automatic control in 2006, both from the Technical University of Catalonia, Barcelona, Spain. Since 2002, he has been an Assistant Professor at the Department of Automatic Control, Technical University of Catalonia. He has been involved in research on artificial intelligence from 1999 to 2002 and, since 2000, on the impact of real-time systems on control systems. His research interests include artificial intelligence, real-time control systems, and collaborative control systems, especially on redundant controllers and multiple controllers with self-interacting systems.

Mohamed El Mongi Ben Gaid was born in Tunis, Tunisia, in 1978. He received the Dipl.-Ing. degree in electrical engineering from the National School of Engineers of Tunis, Tunisia, in 2002, the M.S. degree in distributed computing from the University of Paris XI, Orsay, France, in 2003, and the Ph.D. degree in control and computer engineering from the University of Evry Val dEssonne, Evry, France, in 2006. He is currently a R&D Engineer in real-time computing at the Technology, Computer Science and Applied Mathematics Division of IFP New Energy, the French national research center in energy, transportations and environment. Prior to this, he spent one year as a Postdoctoral Researcher at the Networked Control Systems project-team of INRIA Rhône-Alpes. His research interests include real-time control systems, process monitoring, and multidomain simulation.