Interval Type-2 Locally Linear Neuro Fuzzy Model Based on Locally Linear Model Tree Zahra Zamanzadeh Darban1 and Mohammad Hadi Valipour2() 1

Department of Computer Engineering and Information Technology, Amirkabir University of Technology, Tehran, Iran 2 Research Center for Developing Advanced Technologies, Tehran, Iran [email protected], [email protected]

Abstract. In this paper a new interval Type-2 fuzzy neural network will be presented for function approximation. The proposed neural network is based on Locally Linear Model Tree (LOLIMOT) which is a fast learning algorithm for Locally Linear Neuro-Fuzzy Models (LLNFM). In this research, main measures are to be robust in presence of outlier data and be fast in refining steps. The proposed combination between LOLIMOT learning algorithm and interval type 2 fuzzy logic systems presents a good performance both in robustness and speed measures. The results show that the proposed method has good robustness in presence of noise as we can see in experiments conducted using corrupted data. Also this method has eligible speed as it can be seen in the results. Keywords: LLNFM · LOLIMOT · Type-2 fuzzy systems · Interval Type-2 fuzzy sets

1

Introduction

Recently, studying on Type-2 fuzzy inference systems and Type-2 fuzzy neural networks have been conducted [1]. Type-2 Fuzzy Logic Systems (FLSs) are extensions of Type-1 FLSs [2] that provide more degrees of freedom through uncertainty on Type-1 fuzzy membership. Type-2 fuzzy sets are difficult to understand and utilization because of the three-dimensional fuzzy sets. Also, it is computationally more complicated than using Type-1 fuzzy sets, but it has some advantages such as learning, adaptation, fault-tolerance and generalization [3]. All operations of interval Type-2 fuzzy rules defined by Zadeh’s [2] extension’s principle that can be derived from a set of Type-2 fuzzy rules. A Type-2 membership has grade which can be any subset in [0, 1], on the primary membership [4]. Indeed, for each primary membership, there is a secondary membership that represents the possibilities for the primary membership. To simplify the computation of construction Type-2 fuzzy rules, the secondary membership functions (MFs) can be set to either zero or one and called interval Type-2 sets. Researches on the area of fuzzy rule extraction of Type-2 fuzzy sets are limited. Our main objective is to develop an interval Type-2 fuzzy neural networks based on the construction of Type-2 fuzzy sets with LOLIMOT learning method. The Interval © Springer International Publishing Switzerland 2015 L. Rutkowski et al. (Eds.): ICAISC 2015, Part I, LNAI 9119, pp. 294–304, 2015. DOI: 10.1007/978-3-319-19324-3_27

Interval Type-2 Locally Linear Neuro Fuzzy Model based on Locally Linear Model Tree

295

Type-2 fuzzy neural network (IT2FNN) is a multi-layer network with interval Type-2 fuzzy signals and interval Type-2 fuzzy weights, Gaussian, generalized bell and sigmoid transfer function with principles of Type-2 fuzzy inference system. Briefly the proposed method is an Interval Type-2 Locally Linear Neuro Fuzzy Model based on Locally Linear Model Tree (IT2LL). This paper is organized as follows. In Section 2 and 3, we explain LOLIMOT and interval Type-2 fuzzy logic system respectively. In Section 4 the structure of the proposed Type-2 fuzzy neural network and rule extraction will be declared in details. Simulation and experimental results of function approximation are presented in Section 5. Concluding comments are included in Section 6.

2

Locally Linear Model Tree (LOLIMOT)

2.1

Structure

The basic network structure of a locally linear neuro fuzzy model is depicted in Fig. 1. Every neuron consists of a local linear model which called LLM, and a validity function Φ, which defines the validity of the LLM within the input space. Locally linear neuro-fuzzy models (LLNFM) with locally linear model tree (LOLIMOT) learning algorithm has been introduced in [5]. The network output is calculated as a weighted sum of the outputs of the local linear models, where the validity function is interpreted as the operating point dependent weighting factors. The validity functions are typically chosen as normalized Gaussians. The most important factor for the success of LOLIMOT is divide and conquers strategy that is used in it. The LOLIMOT is an incremental tree construction algorithm, which divides the input space axes in an orthogonal way. In Fig. 2 the LOLIMOT algorithm for the first five iteration steps with a two dimensional input space is depicted. 2.2

Learning Algorithm

In this section, a mathematical formulation of LLNFM with LOLIMOT learning algorithm is described [5]. The fundamental approach with locally linear neuro-fuzzy models divides the input space into small subspaces with fuzzy validity functions. Any produced linear part with its validity function can be described as a fuzzy neuron. Thus the whole model is a neuro-fuzzy network with one hidden layer and a linear neuron in the output layer. The network structure is shown in [5]. In (1) input-output relation of LLNFM is presented. In this formula i is number of neurons, u is the model input, p is number of input dimension, N equals the number of input samples and w denotes the weights of the neuron [5].

yˆ i = w thus

i0

+ w i 1u 1 + ... + w ip u p yˆ =

M

∑y i =1

i

Φ i (u )

(1)

296

Z. Zamanzadeh Darb ban et al.

Fig. 1. Network structure of o a static local linear neuro-fuzzy with neurons for input u [5]]

The validity functions are a chosen as normalized Gaussians. Normalization is necessary for a proper interpreetation of validity functions. In (2) validity function form mulation is represented. μ (u ) where Φi (u ) = M i μ j (u ) ∑ j =1 (2) (u p −cip )2 (u1 −ci 1)2 μi (u ) = exp( ) ×...× exp( ) −2σi21 −2σip2 Each Gaussian validity function f has two parameters: center and standard deviattion .Also there are weight parameters of the nonlinear hidden layer. Optimizationn or learning methods are used to t adjust fine tuning of two sets of parameters, weights and the parameters of validity fu unctions. Local optimization of lin near parameters is simply obtained by Least Squares teechnique. The global parametter vector contains elements. In (3) these parameters are shown. w = [w10 w11 ...w1p... wM 0 ...wMp ] (3) Associated regression matrix m for measured data samples is formulated in (4). T Thus the weights will be obtained d by solving (5) as shown in (6). X = [X 1 X

2

⎡1 u1 (1) ⎢1 u (2) 1 X i =⎢ ⎢# # ⎢ u 1 ( 1 N ) ⎣

...X

M

]

u p (1) ⎤ u p (2) ⎥ ⎥ " # ⎥ ⎥ " u p (N ) ⎦ " "

(4)

Interval Type-2 Locally Linearr Neuro Fuzzy Model based on Locally Linear Model Tree

297

yˆ = X Q wˆ ⎡ Φ i (u (1)) ⎢ 0 Qi = ⎢ ⎢ # ⎢ 0 ⎣

0 Φ i (u (2)) #

" " "

0

"

⎤ ⎥ ⎥ ⎥ ⎥ Φ i (u ( N )) ⎦

wˆ = ( X T QX ) −1 X T Qy

0 0 #

(5)

(6)

Tree based methods are appropriate because of their simplicity and intuitive cconMOT is an incremental based on three iterative steps: ffirst structive algorithm. LOLIM the worst Local Linear Mod del is defined according to local loss functions. This L LLM neuron is selected to be div vided. In the second step all divisions of this LLM on innput space are constructed and checked. c Finally the best division for the new neuron m must be added. For further inform mation about LOLIMOT algorithm refer to [5].

Fig. 2. Operation of the LOL LIMOT in the first five iterations for a two dimensional innput space [5]

In Fig. 2 the first five iteerations of LOLIMOT algorithm for a two dimensionall input space is shown. The co omputation complexity of LOLIMOT grows linearly w with number of neurons. This computation complexity is comparable with other allgorithms. The remarkable pro operties of locally linear neuro-fuzzy model, its transparrency and intuitive construction, lead to the use of Least Squares for rule anteceddent parameters and incrementall learning procedures for rule consequent parameters.

298

3

Z. Zamanzadeh Darb ban et al.

Interval Type-2 Fuzzy Logic System

An Interval Type-2 fuzzy set can represent and handle uncertain information efffectively. That is, interval Ty ype-2 fuzzy sets let us model and minimize the effectss of uncertainties in rule-base interval i Type-2 fuzzy logic systems (IT2FLS). A geneeral interval Type-2 fuzzy logicc system is depicted in Fig. 3. An IT2FLS is very similaar to a type-1 fuzzy logic system ms (FLS) [6] the major structural difference being that the defuzzifier block of type-1 FLS is replaced by the output processing block in an innterval Type-2 FLS, which con nsists of type-reduction followed by defuzzification.

Fig. 3. 3 General Type-2 fuzzy logic system [6]

In Interval Type-2 fuzzy seet, an IT2 FS is bounded from the above and below by ttwo T1 FSs, A and A , which h are called upper MF (UMF) and lower MF (LMF), respectively. The area betweeen A and A is the footprint of uncertainty (FOU) [6]. In practice the computattions in an IT2FLS can be significantly simplified. Consider the rule base of an IT2FL LS consisting of M rules assuming the following form: R

m

x 1 is A1m and … and

: IF

xI

is A pm

m = 1, 2,..., M

(7)

Assume the input vectorr is x ' = ( x1' , x 2' ,..., x 'p ) . So we can compute the meembership of

x i' on each Aim . In (8) the firing interval of mth rule, Fm(x’) is Computeed. F m ( x ' ) = [ μ A m ( x 1' ) × .. × μ A m ( x p' ), μ A m ( x 1' ) × .. × μ A m ( x p' )] 1

m = 1, 2,..., M

4

1

1

1

(8)

Proposed Metho od

In this section, we explain ned our proposed interval Type-2 fuzzy neural netw work which is 3 layer network baased on local linear neuro-fuzzy. The operation of the eeach layer is described as follows. Layer 1: As realized from Fig. 4. this layer is the input layer of the network. E Each input has p dimensions with h crisp value entered to layer 2. According to this, eachh of N input to network represen nts as follow:

Interval Type-2 Locally Linearr Neuro Fuzzy Model based on Locally Linear Model Tree

u (i ) = [1 u 1 (i ) … u p (i )]

i = 1, 2,..., N

299

(9)

Layer 2: This layer con ntains M neuron which are the main parts of the netwoork. Each neuron consists of a LLM, and a validity function Φ which are the outputt of each rule. So each neuron has h two output, O2-1 is output of its rule and O2-2 whichh is output of LLM.

Fig. 4. Inteerval Type-2 locally linear neuro-fuzzy model

To make each rule interv val, each dimension of the rule should become interval.. To do this, we change all p MF Fs of the rule according to (10) which i=1, …, M. [0 MFi + α] ⎧ if MFi − α < 0 ⎪ if MF + α > 1 [MFi − α 1] ⎨ i ((10) ⎪else [MFi −α MFi + α] ⎩ Where α is a value in ran nge (0 1). After that we can calculaate the firing power of each rule: [ μ il

p

μ ir ] = ∏ [ MF jl j =1

MF jr ]

((11)

As the rules are intervall and defined by UMF and LMF, The validity function for any input u(i) is interval which w is shown by Φil and Φir. These validity functions are Gaussian and normalized fo or vector input u as formulated in (12).

300

Z. Zamanzadeh Darban et al.

Φil (u ) =

μil ( u ) M

∑μ

jl

(u)

j =1

Φir (u ) =

μir ( u ) M

∑μ

jr

(12)

(u)

j =1

O 2−1 = [Φil (u ) Φir (u )] Thus the weights which obtained by (6) are interval and we have wil and wir for each rule. According to the calculated weights, the output of each LLM is interval as shown in Fig. 4. The output yi of each neuron should be computed by (13). yˆ il = w il 0 + w il 1u 1 + ... + w ilp u p yˆ ir = w ir 0 + w ir 1u 1 + ... + w irp u p O 2 − 2 = [ yˆ il

(13)

yˆ ir ]

Layer 3: The output sets of the LLM and validity functions are Type-2 fuzzy sets. To obtain embedded type-1 fuzzy sets for each neuron, a type-reduced method, centroid type reduction, is used in type-reducer. The efficient algorithm, called the Karnik-Mendel (KM) algorithm [7], has been developed for centroid type reduction. KM Algorithm get validity functions (Φil and Φir) and output of LLMs (yil and yir) as input. In this way, it reduce type of output to type-1 fuzzy and calculate output of the Type-2 LLNFM simultaneously. The output of the network is: ~

O 3 = y = KM ( y l , y r , Φ l , Φ r )

4.1

(14)

Structure Learning

There are no general guidelines that can be applied to specify the optimal number of fuzzy rules and its corresponding initial values for the FNN. In this study, we propose a Type-2 LOLIMOT method which consists of incremental learning procedures, linear regression, and fuzzy rule extraction to solve this problem. The flowchart of the structure identification for The Type-2 fuzzy static local linear neuro-fuzzy and the process is depicted in Fig. 5. LOLIMOT convert the worst rule of the existing rules to 2 rules via dividing it in a dimension which is caused that rule worst. Changing LOLIMOT method working to Type-2 fuzzy, we change all membership functions of rules to interval Type-2 fuzzy which intervals have the equal size. Therefore, we have Type-2 fuzzy rules and in each iteration. The interval Type-2 local linear neuro-fuzzy has a network structure as shown in Fig. 4. The brief description of the functionality and the operation of the whole neural networks is as follows:

Interval Type-2 Locally Linear Neuro Fuzzy Model based on Locally Linear Model Tree

301

The input data enter to fuzzification layer which contains neuron realizes a LLM and M interval rules with an associated validity function that determines the region of validity of the LLM. To change LOLIMOT to Type-2, Adding new rules with LOLIMOT and learning of the Adding new rules with LOLIMOT and learning of the 2 fuzzy, construct the validity functions for the beginning input space. Set M = 1 and start with one LLM, which covers the whole input space with the interval uncertainty. Then, find the worst Type-2 fuzzy rule with calculating the loss function for each of the local linear models (LLM). A local loss function is used, with a squared output error: L Fi =

N

∑e

2

( j ) * ( Φ il + Φ ir ) * (u ( j ))

j =1

(15)

The worst LLM is maxi(LFi) where i=1, … M. Training Data

No

Find rules with type-2 lolimot

Obtained Weights of each rule

Calculate validity functions

Calculate yil and yir for each LLM

KM Type Reducer

Desired RSME?

Yes

Output

Fig. 5. Structure identification for learning interval Type-2 locally linear neuro-fuzzy model

After finding the worst LLM, check all half division in every p dimension. To select the best division and relocation of two new rules with the worst rule, we should calculate validity functions of each rules obtained by division. The best among p alternatives is chosen based on generating minimum mean square error (mse) of the network output, because our target is to achieve desired mse with minimum number of rules. Then, the validity functions will be added to the model and the number of LLM is incremented to M + 1. Adding new rules by LOLIMOT and learning of the local linear neuro-fuzzy is continued until the mse is reached to the desired mse or don’t change in several iterations.

302

5

Z. Zamanzadeh Darban et al.

Experiments and Results

We have conducted 6 experiments using 6 non-linear functions, in order to test the proposed method. These functions are described as below in equations (16) to (21).

f1 ( x) = 3x( x − 1)( x − 1.9)( x − 0.7)( x + 1.8) f 2 ( x) = 10[sin( 4 x + 0.1) + sin(14 x) + sin(10 x + 0.2) + sin(17 x + 0.3)

⎛ 2000( x − 0.1)( x − 0.3)( x − 0.5) ⎞ ⎜ ⎟ ⎜ × ( x − 0.9)( x − 1.1)( x + 0.2) ⎟ ⎜ × ( x + 0.4)( x + 0.6)( x + 0.8)( x + 1) ⎟ ⎟ f3 ( x) = 5 tan −1⎜ x 2 + 1.5 x + 1 ⎜ ⎟ ⎜ ⎟ ⎜⎜ ⎟⎟ ⎝ ⎠

f 4 ( x1 , x2 ) = 3x1 ( x1 − 1)( x1 − 1.9)( x1 − 0.7) × ( x1 + 1.8) sin( x2 ) 1 f 5 ( x1 , x2 ) = f 2 ( x1 ) 10 2 2 2 ⎛ x −0.8 ⎞ ⎞ ⎛ x + 0.75 ⎞ ⎛ x − 0.1 ⎞ ⎛ −⎜ −⎜ −⎜ ⎟ ⎟ ⎟ ⎜ 0 . 25 0 . 15 0 . 1 ⎠ ⎟ ⎠ ⎠ × 10e ⎝ − 8e ⎝ − 4e ⎝ ⎜ ⎟ ⎝ ⎠ 2 2 sin(10 x1 + 5 x2 − 6 x2 ) f 6 ( x1 , x2 ) = 10 10 x12 + 5 x22 − 6 x2

(16)

(17)

(18)

(19)

(20)

(21)

The uncorrupted training dataset consist of randomly of 300 generated instances, with corresponding inputs and output of f1 to f6. Also a set of 100 uncorrupted instances, generated in the same way, has been used as testing dataset. A corrupted training instance is composed of the same output as the corresponding uncorrupted one but with the input corrupted by adding a random value from a normal distribution with zero mean and standard deviation σ=0.1. In these experiments, three corrupted dataset are used, in which 10%, 20% and 30% of the instances are randomly corrupted. Samples of the test functions are shown in Fig. 6. In all of the experiments, for fair comparison between methods, the same number of rules are used in simulations. Methods, which are used in respective experiments are LLNFM based on LOLIMOT (T1LL), Type 2 Interval LLNFM based on LOLIMOT (IT2LL) and TSK Fuzzy Neural Network (T1FNN) [8].

Interval Type-2 Locally Linear Neuro Fuzzy Model based on Locally Linear Model Tree

0 x

-50 -1

2

20

f5(x 1,x 2)

f4(x 1,x 2)

-50 -2

0 -20 2 0 x1 -2 -2

2

0 x2

0

0 x

20 0 -20 1 0 x 1 -1 -1

0 x2

1

0

-10 -1

1

f6(x 1,x 2)

0

10 f3(x)

50 f2(x)

f1(x)

50

303

0 x

1

10 5 0 -5 1 0 x1 -1 -1

0 x2

1

Fig. 6. Samples of the test functions. Table 1. Experiment Results for f1, f2, f3, f4, f5 and f6 f1

f2

tr te tr te IT2LL 0.0019 0.0018 0.0032 0.0032 T1LL 0.0043 0.0046 0.0071 0.0070 T1FNN 0.0014 0.0014 0.0028 0.0030 f1 f2 tr te tr te IT2LL 0.0741 0.0204 0.1225 0.0370 T1LL 0.0825 0.0311 0.1379 0.0454 T1FNN 0.0799 0.0234 0.1675 0.0526 f1 f2 tr te tr te IT2LL 0.1066 0.0412 0.1843 0.0767 T1LL 0.1431 0.0891 0.2245 0.1361 T1FNN 0.1092 0.0584 0.2211 0.1265 f1 f2 tr te tr te IT2LL 0.1158 0.0530 0.2007 0.0906 T1LL 0.1982 0.1519 0.3170 0.2346 T1FNN 0.1139 0.0983 0.2242 0.2190

uncorrupted data f3 f4 tr te tr te 0.0089 0.0088 0.0253 0.0258 0.0114 0.0118 0.0647 0.0662 0.0070 0.0066 0.0283 0.0299 10% corrupted data f3 f4 tr te tr te 0.3507 0.1023 0.0438 0.0273 0.2206 0.0808 0.0943 0.0768 0.4042 0.1092 0.0513 0.0414 20% corrupted data f3 f4 tr te tr te 0.5034 0.1925 0.0569 0.0281 0.3773 0.2307 0.1134 0.0813 0.5477 0.2873 0.0626 0.0571 30% corrupted data f3 f4 tr te tr te 0.5575 0.2640 0.0702 0.0292 0.5065 0.3796 0.1287 0.1001 0.5907 0.4856 0.0809 0.0685

f5 f6 tr te tr te 0.0685 0.0613 0.0292 0.0275 0.1127 0.1019 0.0795 0.0773 0.0634 0.0690 0.0332 0.0376 f5 f6 tr te tr te 0.1128 0.0652 0.0496 0.0287 0.1678 0.1213 0.1210 0.0870 0.1100 0.0994 0.0574 0.0507 f5 f6 tr te tr te 0.1581 0.0635 0.0669 0.0297 0.1924 0.1250 0.1430 0.0972 0.1342 0.1320 0.0746 0.0737 f5 f6 tr te tr te 0.1840 0.0684 0.0790 0.0313 0.2139 0.1539 0.1605 0.1112 0.1806 0.1534 0.0902 0.0838

The results are shown in Table 1 and Table 2, where the training and testing error (abbreviated with “tr” and “te”, respectively) are root mean square error (RSME). As it can be seen in Table 1 and Table 2, IT2LL estimates are better than T1LL and also are comparable with T1FNN results. For corrupted data with progressively increased corruption, IT2LL is more robust to input space outliers. The results show

304

Z. Zamanzadeh Darb ban et al.

that the outliers outperform m T1LL and T1FNN estimates. It seems that T1LL cannnot handle the noise and T1FNN N may overfit the training data. On the other hand theree is a comparison between refining times of these thhree methods. As it shown in Fig. F 7, T1LL’s refining time for each function, is less tthan other methods. The refining g time for IT2LL and T1FNN is not very different.

Fig. 7. Reffining time for each function in three methods

6

Conclusion

A new interval Type-2 fuzzzy neural network for function approximation has bbeen introduced in this paper. Th he proposed neural network was a combination of Locally Linear Neuro-Fuzzy Modeel and Interval Type-2 Fuzzy Logic System. The learnning algorithm, which has been modified to use in proposed method, was Locally Linnear Model Tree which is one of o the fast learning algorithm for LLNFM. Regarding the simulation results, it can be b concluded that the proposed method is fast and alsoo is robust in presence of outlierrs.

References 1. Karnik, N.N., Mendel, J.M M., Liang, Q.: Type-2 Fuzzy Logic Systems. IEEE Trans. on Fuuzzy Systems 7(6), 643–658 (19 999) 2. Zadeh, L.A.: Fuzzy Sets. In nformation and Control 8, 338–353 (1965) 3. Wu, D., Mendel, J.M.: Un ncertainty Measures for Interval Type-2 Fuzzy Sets. Informaation Sciences 77(23), 5378–539 93 (2007) 4. Mendel, J.M.: Type-2 Fuzzzy Sets and Sytems: An Overview. Computational Intelligeence Magazine 2(1), 20–29 (200 07) 5. Nelles, O.: Local Linear Model M Tree for On-line Identification of Time Variant Nonlinnear Dynamic Systems. In: Inteernational Conference on Artificial Neural Networks (ICAN NN), pp. 115–120 (1996) I Type-2 Fuzzy Logic Systems: Theory and Design. IE EEE 6. Liang, Q., Mendel, J.M.: Interval Transactions on Fuzzy Systtems 8, 535–550 (2000) 7. Karnik, N.N., Mendel, J.M M.: Centroid of a Type-2 Fuzzy Set. Information Sciences 132(14), 195–220 (2001) 8. Takagi, T., Sugeno, M.: Fuzzy F identification of systems and its applications to modell-ing and control. IEEE Trans. Systems, Man, Cybernetics SMC 15(1), 116–132 (1985)