Experimental Evaluation of some Thresholding Methods for Estimating Time-Delays in Open-Loop

Experimental Evaluation of some Thresholding Methods for Estimating Time-Delays in Open-Loop Svante Bj¨orklund Control & Communication Department of ...
2 downloads 0 Views 485KB Size
Experimental Evaluation of some Thresholding Methods for Estimating Time-Delays in Open-Loop

Svante Bj¨orklund Control & Communication Department of Electrical Engineering Link¨opings universitet, SE-581 83 Link¨oping, Sweden WWW: http://www.control.isy.liu.se E-mail: [email protected] 14th July 2003

OMATIC CONTROL AU T

CO MM U

NICATION SYSTE

MS

LINKÖPING

Report no.: LiTH-ISY-R-2525

Technical reports from the Control & Communication group in Link¨oping are available at http://www.control.isy.liu.se/publications.

Datum Date

Avdelning, Institution Division, department

Automatic Control Department of Electrical Engineering

Språk Language

X

Rapporttyp Report: category

Svenska/Swedish Engelska/English

X

Licentiatavhandling Examensarbete C-uppsats D-uppsats Övrig rapport

URL för elektronisk version

2003-07-15

ISBN ISRN Serietitel och serienummer Title of series, numbering

ISSN

1400-3902

LiTH-ISY-R- 2525

http://www.control.isy.liu.se

Titel Title

Experimental Evaluation of some Thresholding Methods for Estimating Time-Delays in Open-Loop

Författare Author

Svante Björklund

Sammanfattning Abstract

In this report we study estimation of time-delays in linear dynamical systems with additive noise. Estimating time-delays is a common engineering problem, e.g. in automatic control, system identification and signal processing. The purpose with this work is to test and evaluate a certain class of methods for time-delay estimation, especially with automatic control applications in mind. Particularly interesting it is to determine the best method. Is one method best in all situations or should different methods be used for different situations? The tested class of methods consists essentially of thresholding the cross correlation between the output and input signals. This is a very common method for timedelay estimation. The methods are tested and evaluated experimentally with the aid of simulations and plots of RMS error, bias and confidence intervals.

95-11-01/lli

The results are: The methods often miss to detect because the threshold is too high. The threshold has nevertheless been selected to give the best result. All methods over-estimate the time-delay. Nearly the whole RMS error is due to the bias. None of the tested methods is always best. Which method is best depends on the system and what is done when missing detections. Some form of averaging of the cross correlation, e.g. integration to step response or CUSUM, is advantageous. Fast systems are easiest. White noise input signal is easiest and steps is hardest. The RMS-errors are high in average (approximately greater than 6 sampling intervals). The error is lower for fast system or for high SNR.

Nyckelord Keywords

time-delay, dead-time, estimation, system identification, linear dynamic systems, cross correlation, ANOVA, confidence intervals, simulations

2

Contents 1 Introduction

1

2 Thresholding methods

2

3 Simulation setup

3

4 Analysis

6

4.1

Simple analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

6

4.2

Statistical analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

8

5 Discussion and conclusions

16

5.1

Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

5.2

Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

5.3

Recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

5.4

Future work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

References

1

A Validation of confidence intervals

1

1

2

1

Introduction

The problem we address in this report is estimating time-delays in linear dynamical systems with additive noise. A synonym for time delay is dead-time. Estimating time-delays is a common engineering problem, e.g. in control performance monitoring of industrial processes [Hor00, Swa99], in design and tuning of controllers, in range estimation in radar [KQ92] and in direction estimation by time-delay of arrival in signal intelligence [HR97, Wik02]. Dead-time estimation is also a necessary part in all system identification [Lju99]. The purpose with this work is to test and evaluate a certain class of methods for timedelay estimation, especially with automatic control applications in mind. Particularly interesting is it to determine the best method. Is one method best in all situations or should different methods be used for different situations? The tested class of methods consists essentially of thresholding the cross correlation between the output and input signals. This is a very common method for time-delay estimation. The methods are tested and evaluated experimentally with the aid of simulations and plots of RMS error, bias and confidence intervals. In the next chapter, the thresholding methods are briefly described. Then, Chapter 3 is about the simulation setup. After that, in Chapter 4 the analysis of the simulations is conducted. Following, Chapter 5 contains discussion, conclusions and suggestions for further work. After a literature reference list, Appendix A contains validation of required prerequisites for the analysis.

1

2

Thresholding methods

Thresholding methods is a subgroup of cross correlation methods. The steps of the methods are: 1. Estimate the impulse response and estimate the uncertainty of the impulse response estimate. The estimated impulse response is in principle the cross correlation between the output and input signals of the system [Bj¨o03]. 2. Optionally, integrate to step response. 3. Thresholding. If the number zero is outside a certain confidence interval, then we consider the impulse (step) response to have started and this point of time is the time-delay estimate. The thresholding can be either • Direct thresholding [Bj¨o03] or.

• Cumulative sum (CUSUM) thresholding [Bj¨o03]. For information about CUSUM see also [Gus00, GLM01].

There are some method parameters to choose. The most important are • The relative threshold hstd and relative drift νstd [Bj¨o03]. The methods that we have used in this report are: • idimp5. Direct thresholding of impulse response with prewhitening and hstd = 5. This would give a confidence interval with a confidence level of 0.999999713 if the impulse response estimates are Gaussian distributed (which is a good assumption, see [Lju99]) and the estimate of the uncertainty in the confidence interval estimate is accurate. • idstep5. Direct thresholding of step response without prewhitening and h std = 5. • idimpCusum4. CUSUM thresholding of impulse response with prewhitening, h std = 3, νstd = 1 . • idstepCusum4. CUSUM thresholding of step response with prewhitening, h std = 1, νstd = 6 . The choices of hstd and νstd are the result of a statistical analysis in [Bj¨o03]. Note the opposite relation between hstd and νstd for idimpCusum4 and idstepCusum4. For more on the used methods in this report see [Bj¨o03]. In signal processing applications the system often consists of a pure time delay, maybe with an amplitude change. Then the right thing to do is to estimate the peak of the impulse response (cross correlation) instead. 2

3

Simulation setup

The setup for the simulations is the same as in [Bj¨o03] but with different time-delay estimation methods. Since the estimates are very non-Gaussian, so many as 4096 trials were simulated. From this 4 estimates of RMS error was calculated for the plots with confidence intervals. In this report we are studying properties in average. There were many missed detections because the threshold was often not crossed. This complicates the analysis. It is not obvious what to do when missing. One possibility is to assign a value to the time-delay estimate. If this value is very incorrect, then we give a penalty because of the miss. Another possibility is to give an alarm. A third is to find the highest peak of the impulse response. This latter possibility will give a bias in the estimate. It, unfortunately, turns out that which method is the best depends on what we do when the detection is missed. In this report, when a method missed to detect a uniform distributed random number in the range 20 to 30 was delivered as the time-delay estimate (as in Section 7.7-7.9 in [Bj¨o03]). The reason for this special choice of was to come closer to the required prerequisites for the confidence interval calculations. See appendix A and reference [Bj¨o03] for more on this subject. Three environment factors were varied during the simulations: The system, the input signal type and the SNR [Bj¨o03]. The SNR was either 1 or 100. See [Bj¨o03] for the definition of the SNR. The impulse responses of the four used systems are depicted in Figure 3.1. Note that for all the systems the time delay will be 10 after the sampling. More information about the systems can be found in [Bj¨o03]. Figures 3.2-3.4 show the used input signals in the time and frequency domains. More information about the input signals can be found in [Bj¨o03].

3

Impulse response G1 (t130g)

Impulse response G2 (t130g)

From: U(1)

0.08

0.04

To: Y(1)

0.6 Amplitude

To: Y(1)

Amplitude

0.06

0.02

0

From: U(1)

0.8

0.4

0.2

0

14

28

42

56

0

70

0

14

28

42

Time (sec.)

Time (sec.)

Impulse response G5 (t130g)

Impulse response G6 (t130g)

From: U(1)

0.08

56

70

56

70

From: U(1)

0.05 0.04

Amplitude

0.04

0.03 To: Y(1)

To: Y(1)

Amplitude

0.06

0.02 0.01

0.02 0 0

0

14

28

42

56

−0.01

70

0

14

Time (sec.)

28

42

Time (sec.)

Figure 3.1: Impulse response of system G1 -G2 and G5 -G6 . True time-delay after sampling Td = 10.

Time signal RBS10−30%

Power spectruml RBS10−30%

1

10

1 0.8 0.6 0

0.4

10

Power

0.2 0 −0.2 −1

−0.4

10

−0.6 −0.8 −1 −2

0

50

100

150

200

250 300 samples

350

400

450

500

10

0

0.1

0.2

0.3

0.4 0.5 0.6 Normalized frequency

0.7

0.8

0.9

1

Figure 3.2: Time signal (left) and frequency spectrum (right) for a realization of the input signal type RBS 10-30%.

4

Time signal RBS0−100%

Power spectruml RBS0−100%

1

10

1 0.8 0.6 0.4

Power

0.2 0

0

10

−0.2 −0.4 −0.6 −0.8 −1 −1

0

50

100

150

200

250 300 samples

350

400

450

10

500

0

0.1

0.2

0.3

0.4 0.5 0.6 Normalized frequency

0.7

0.8

0.9

1

Figure 3.3: Time signal (left) and frequency spectrum (right) for a realization of the input signal type RBS 0-100%.

Time signal steps

Power spectruml steps

2

10

1 1

0.8

10

0.6 0

0.4

10

Power

0.2 0

−1

10

−0.2 −2

−0.4

10

−0.6 −3

−0.8

10

−1 −4

0

50

100

150

200

250 300 samples

350

400

450

500

10

0

0.1

0.2

0.3

0.4 0.5 0.6 Normalized frequency

0.7

0.8

0.9

1

Figure 3.4: Time signal (left) and frequency spectrum (right) for a realization of the input signal type Steps.

5

4

Analysis

In this chapter we compare the tested methods and see in which cases they could be used. In Section 4.1 we start by a simple analysis that can give some feeling about the problems and how the methods work. Section 4.2 then has a statistical analysis, from which it is possible to draw conclusions.

4.1

Simple analysis

Figure 4.1 displays estimated impulse response for one of the systems but for different input signals and different SNRs. We see that in some cases the estimate is very inaccurate. This makes the time-delay estimation a hard job in these cases. Impulse response ,10−30%, SNR=100, G1

0.08 0.06

0.1

0.04

0.05

0.02

0

0

−0.05

−0.02

0

10

0.08

Impulse response ,10−30%, SNR=1, G1

0.15

20

30 40 50 60 Time [s] Impulse response ,0−100%, SNR=100, G1

−0.1

70

0

10

0

10

0.15

0.06

20

30 40 50 Time [s] Impulse response ,0−100%, SNR=1, G1

60

70

60

70

60

70

0.1

0.04 0.05 0.02 0

0 −0.02

0

10

20

30

40

50

60

−0.05

70

Time [s] Impulse response ,steps, SNR=100, G1

0.3

30

40

50

Time [s] Impulse response ,steps, SNR=1, G1

2

0.2

20

1

0.1 0 0 −1

−0.1 −0.2

0

10

20

30 40 Time [s]

50

60

−2

70

0

10

20

30 40 Time [s]

50

Figure 4.1: Impulse response estimate of the system G1 by the function idimp4 for different input signal types and different SNRs [Bj¨o03]. The solid line is the true impulse response. The circles are the estimated impulse response and the triangles mark ±two estimated standard deviations. Note the different ranges of the vertical axes. (t130f1.m) Direct thresholding of the impulse response estimate is illustrated in Figure 4.2 and CUSUM thresholding in Figure 4.3. Note that in these figures, the relative threshold h std and relative drift νstd were not the same as in Chapter 2 and in the statistical analysis in Section 4.2.

6

Threshold test

Impulse response

0.12

0.09

0.1

0.08

0.08

0.07

0.06

0.06 0.04

0.05 0.02

0.04 0

0.03

−0.02

0.02

−0.04

0.01

−0.06 −0.08

0

10

20

30

Time [s]

40

50

60

0

70

0

10

20

30 40 Time [samples]

50

60

70

Figure 4.2: Left: Estimated impulse response with uncertainty. Right: Estimated impulse response and threshold. Simulated input signal of type RBS 10-30%. SNR was 1. System G 1 . The estimated time delay with idimp4 (hstd = 3) was Tˆd = 11. (t134d1.m)

CUSUM test statistics G

Impulse response G

1

0.12

1

0.07

0.1

Test statistics threshold drift

0.06

0.08

0.05 0.06

0.04

0.04

0.02

0.03

0

0.02 −0.02

0.01

−0.04

−0.06

0

10

20

30

Time [s]

40

50

60

70

0

0

10

20

30 40 Time [samples]

50

60

70

Figure 4.3: Left: Impulse response with uncertainty. Right: Test statistics g(t), threshold h and drift ν for CUSUM on impulse response. Simulated input signal of type RBS 10-30%. SNR was 1. System G1 . The estimated time delay with idimpCusum3 (hstd = 2 and νstd = 1) was Tˆd = 11. (t146b1.m)

7

4.2

Statistical analysis

In this section we will perform a statistical analysis. Bear in mind that: • All results are in average. Certain special cases can give a different result. • Even if there is a statistically significant difference, it is not sure that the difference has any practical importance. The statistical analysis is conducted in the same way as in Chapter 7 in [Bj¨o03]. The transformation was (RMS error)ˆ(0.73028) (See [Bj¨o03]). This means “The lower the better”. Figure 4.4 shows confidence intervals for pair-wise comparisons of methods. We see that: • Step response is significantly better (not overlapping confidence intervals) than impulse response in average. • Step response: no significant difference (overlapping confidence intervals) between direct and CUSUM thresholding. • Impulse response: significant difference between direct and CUSUM thresholding and CUSUM is better. t149b5:Method 1:idimp5 2:idstep5 3:idimpCusum4 4:idstepCusum4

1

2

3

4

3.5 4 4.5 3 groups have population marginal means significantly different from Group 1

Figure 4.4: Confidence intervals (the lines in the circles) for pair-wise comparisons (95% simultaneous confidence level) for different thresholding methods. Positive transformation: (RMS error)ˆ(0.730028) => ”The lower the better”. (t149b1.m) Figure 4.5 shows the average RMS error and bias for the different methods. Wee see that the RMS error ' 6 in average. It is high because: 8

• We are punishing missed detections. • The case with SNR=1 is difficult difficult. • The methods are overestimating the time delay. Nearly the whole RMS error is due to the bias. t149b5:030505 17:30 rms: rms, data(:,m,m,m,:,m,m,m,m)

t149b5:030505 17:30 bias: bias, data(:,m,m,m,:,m,m,m,m)

. MAX

. MAX

8.67

8.39

. MIN . MIN

6.06

4.47 7.36

idimp5

idimp5

6.69

10

10

8

8 idstep5

6.32

6

idstep5 6

4

5.42

4 idimpCusum4

2

idimpCusum4

2

0

0 idstepCusum4

Method

idstepCusum4



Method



Figure 4.5: Average RMS error (left) and bias (right) of time-delay estimates for different thresholding methods. In Figure 4.6 it is obvious there is a large difference in performance between low and high SNR. Using step response estimates, the mean RMS error is 2.0 for high SNR but 10.4 for low SNR. Nearly the whole RMS error is due to the bias. In Figure 4.7 wee see how good the methods are for different input signal types. We notice: • No method is always significantly best but step response is often better than impulse response. • Wideband random signal easiest for all methods. • Steps signal hardest for all methods. Figure 4.8 shows the average RMS error and bias for different methods and input signal types. Wee see that: • For the wideband random signal, RMS error ≈ 3.6 using step response. • For steps as signal, RMS error ≈ 9.5 using step response. In Figure 4.9 wee see how good the methods are for different input signal types. We notice: 9

t149b5:030505 17:30 rms: rms, data(:,m,m,:,m,m,m,m,m)

t149b5:030505 17:30 bias: bias, data(:,m,m,:,m,m,m,m,m)

. MAX

. MAX

12.9

12.5

9.94 11.8

14

14

12

7.84

10.9

12

10

10

10.9 4.46

8

4.27

9.33

8

6

6

4

. MIN

4 2.18

2 0

2

2.9

idimp5

1.1

0

idimp5

2.46

. MIN 1

1

idstep5

1.78

idstep5 1.51

idimpCusum4

idimpCusum4

100

100 idstepCusum4

idstepCusum4

Method

SNR

Method

SNR

Figure 4.6: Average RMS error (left) and bias (right) of time-delay estimates for different thresholding methods and SNRs.

t149b5:Method*InType

1:idimp5*RBS10−30% 2:idstep5*RBS10−30% 3:idimpCusum4*RBS10−30% 4:idstepCusum4*RBS10−30% 5:idimp5*RBS0−100% 6:idstep5*RBS0−100% 7:idimpCusum4*RBS0−100% 8:idstepCusum4*RBS0−100% 9:idimp5*steps 10:idstep5*steps 11:idimpCusum4*steps 12:idstepCusum4*steps 2

2.5 3 3.5 4 4.5 5 5.5 6 6.5 11 groups have population marginal means significantly different from Group 1

Figure 4.7: Confidence intervals (the lines in the circles) for pair-wise comparisons (95% simultaneous confidence level) for different thresholding methods and input signals. Positive transformation: (RMS error)ˆ(0.730028) => ”The lower the better”. (t149b1.m)

10

t149b5:030505 17:30 rms: rms, data(:,:,m,m,m,m,m,m,m)

t149b5:030505 17:30 bias: bias, data(:,:,m,m,m,m,m,m,m)

6.55

. MAX 15

6.2

5.02

6.02

15

13.4

. MAX

5.85

3.15

13.1

6.06 10

5.63 10

. MIN

. MIN

3.39 5

5.87

9.76

5

9.37

0.887

4.8 0

11.2

4.31 3.73

0 10.7

idimp5

RBS10−30%

3.77

RBS10−30%

idimp5

2.81

9.32

9.12

idstep5

idstep5 RBS0−100%

RBS0−100%

idimpCusum4

idimpCusum4 steps

idstepCusum4

steps idstepCusum4

InType

Method

InType

Method

Figure 4.8: Average RMS error (left) and bias (right) of time-delay estimates for different thresholding methods and input signals. • Impulse response significantly best for the fast system. • Step response significantly best for the other systems. • The fast system is the easiest for all methods. Figure 4.10 shows the average RMS error and bias for different methods and systems. We see that: • Fast system: RMS error ≈ 3.5 (step response), 2.9 (impulse response). • High order system with complex poles: RMS error ≈ 7.8 (step response), 10.5 (impulse response). Figures 4.11 and 4.12 depict the RMS error and bias of the methods for all environment factors. We observe that: • The bias is most often positive . It is often large. The few cases with negative bias has a very small bias. This indicate that we need a better estimation of the change time point in the impulse and step response.

11

t149b5:Method*Sys

1:idimp5*slow2 2:idstep5*slow2 3:idimpCusum4*slow2 4:idstepCusum4*slow2 5:idimp5*fast2 6:idstep5*fast2 7:idimpCusum4*fast2 8:idstepCusum4*fast2 9:idimp5*real4 10:idstep5*real4 11:idimpCusum4*real4 12:idstepCusum4*real4 13:idimp5*cplx4 14:idstep5*cplx4 15:idimpCusum4*cplx4 16:idstepCusum4*cplx4 1

1.5 2 2.5 3 3.5 4 4.5 5 5.5 14 groups have population marginal means significantly different from Group 1

Figure 4.9: Confidence intervals (the lines in the circles) for pair-wise comparisons (95% simultaneous confidence level) for different thresholding methods and systems. Positive transformation: (RMS error)ˆ(0.730028) => ”The lower the better”. (t149b1.m)

12

t149b5:030505 17:30 rms: rms, data(:,m,m,m,m,m,m,m,:)

t149b5:030505 17:30 bias: bias, data(:,m,m,m,m,m,m,m,:)

10.4

10.1

6.28 4.55 10.6 15

10

7.64

2.88

10.3

6.75

2.52

15

. MAX

3.67

10.8

. MIN

5.55

5.07

3.27 10.2

2.29

10.6

5

7.57 0

10

2.82

8.79

5

. MIN

. MAX

6.56

6.71

slow2

5.97

0

7.35

2.44

7.97

slow2

2.59 9.59 6.33

idimp5

fast2

idimp5

fast2

8.11

7.2

idstep5

idstep5 real4

real4

idimpCusum4

idstepCusum4

idimpCusum4 cplx4

Sys

idstepCusum4

Method

cplx4

Sys

Method

Figure 4.10: Average RMS error (left) and bias (right) of time-delay estimates for different thresholding methods and systems.

13

t149b5:030505 17:30 rms: rms, data(:,:,m,:,m,m,m,m,:)

0.829 15.3 0. 0 0.458 1.45 0.923 15.3 15.2 2.68 15.2 20 10 0

1.22

4.19

15.3

16

1.72

15.3

idimp5*100

6.89

14.1

1.09

idstep5*100

16.2

idimpCusum4*100 idimpCusum4*1 idstepCusum4*100

0

1.13

1.81

6.9

slow2*RBS10−30%

15.3

slow2*RBS0−100% 2.49

0.636 15.5

slow2*steps fast2*RBS10−30%

fast2*RBS0−100%

13.7

fast2*steps

8.46

real4*RBS10−30%

15.4

real4*RBS0−100%

14.2

real4*steps

10

5.66

3.09

0.323

0.637

1.29

12.2

0.88 0.268

15

4.15

0.906

15.3

0.465 15.2

15.4

13.7

14.5

9.62

5.19

0.541

15.6

0.573

0.23

0.241

1.48

idstep5*1

4.56

0.595

1.94

0.981

15.4

16.3

1.17

0.839 11.6 9.02 1.37

. MAX 16.5

idimp5*1

5.65

10.6

0.79

11.1

12.4

2.14 5.74

3.16

1.93

1.32

1.13 1.96

0.109 15.3

2.04

15.4

15.3

1.42

0

15.3

15.3

MIN

15.3

15.3

0.15

cplx4*RBS10−30%

15.4

cplx4*RBS0−100% cplx4*steps

Sys*InType

idstepCusum4*1 Method*SNR

Figure 4.11: Average RMS error of time-delay estimates for different thresholding methods and different environment factors.

14

t149b5:030505 17:30 bias: bias, data(:,:,m,:,m,m,m,m,:)

0.687 15.1

0.0225 15 15.1

0 0

15

0.134

0.309 −0.585

6.68

3.11

2.22 0.3 15.5 0.0117 −0.00171 −0.197 13.6 −0.0522 15 1.24 −0.753 −0.0388 4.53 7.18 0.661 15 0.802 15 0 . MIN −0.771 −0.00708 8.86 0.377 15 14.9 15.5 0.644 −0.0471 0.000977 3.04 5.35 15 2.42 7.58 slow2*RBS10−30% −0.255 15 −0.00732 −0.0361 −0.0403 15 1.14 4.14 14.7 slow2*RBS0−100% 2.98 0.89 −0.104 0.541 15 15 15.3 14.2 1.07 slow2*steps 0.38 0 −0.0186 15 9.12 7.97 fast2*RBS10−30% −0.0266 9.39 15.1 1.05

20 0 −20

5.69

idimp5*100 idimp5*1 idstep5*100

15 4.28 fast2*RBS0−100% 1.26 10.4 0.735 . MAX 15.7 14.9 fast2*steps 0.837 4.11 6.69 13.5 12.2 real4*RBS10−30% 15 1.55 15 real4*RBS0−100% 11.5 1.11

idstep5*1 idimpCusum4*100 idimpCusum4*1 idstepCusum4*100

5.61

real4*steps

8.37

cplx4*RBS10−30%

15.1

cplx4*RBS0−100% cplx4*steps

Sys*InType

idstepCusum4*1 Method*SNR

Figure 4.12: Average bias of time-delay estimates for different thresholding methods and different environment factors.

15

5

Discussion and conclusions

5.1

Discussion

There is a very high confidence level for the direct thresholding. This implies that we overestimate the time delay. All methods over-estimate the time-delay. The used thresholds have nevertheless been selected to give the best result [Bj¨o03]. A better estimation of the change time than simple thresholding is needed. In [KG81] a more sophisticated estimation method of the change time is used. This method has not been tested in this report.

5.2

Conclusions

We draw the following conclusions from the work in this report: • The methods often miss to detect. This is a problem in the analysis. • No method is always best. Which is best depends on the system and what is done when missing detections. • In the tested cases (the used penalty when missing), using step response was best in average. It was also best in most combinations of factor environments. • In the tested cases, using impulse response was best in average for fast systems. • No significant difference between direct and CUSUM thresholding for step response. • CUSUM significantly better than direct thresholding for impulse response except for fast systems. • Some form of averaging, e.g. integration to step response or CUSUM, of the impulse response enhances the time-delay estimate (except for fast systems). Integrating to step response and CUSUM are two ways to smooth the impulse response estimate. Using CUSUM on the step response estimate does not further improve the estimates in most cases. • The wideband random input signal is easiest. Input signal with steps is hardest. • Fast system is easiest. • All methods over-estimate the time-delay. A better estimation of the change time is needed. • Nearly the whole RMS error is due to the bias. • The RMS-errors are high in average (& 6 sampling intervals). This is too high to useful. The error is lower for fast system or for high SNR.

5.3

Recommendations

• Use thresholding methods only for fast systems or when the SNR is high. • For step input the SNR must be high. • Integration to step response and/or CUSUM thresholding enhances the performance for the impulse response estimate, except for fast systems.. 16

5.4

Future work

Some possible future work is: • Estimate better the change point in the impulse and step responses to avoid overestimating the time-delay. • Are there really no better thresholds and drifts? • Test logarithmic cross spectrum scale (cepstrum), see [Bj¨o03]. • Test the thresholding methods on random systems and make a statistical analysis. • Test other methods in the same way as in this report. • Compare groups of methods.

17

18

References [Bj¨o03]

Svante Bj¨orklund. Experimental evaluation of some cross correlation methods for time-delay estimation in linear systems. Technical Report LiTH-ISYR-2513, Department of Electrical Engineering, Link¨oping University, SE-581 83 Link¨oping, Sweden, April 2003.

[GLM01] F. Gustafsson, L. Ljung, and M. Millnert. Signalbehandling. Studentlitteratur, Lund, Sweden, 2001. In Swedish. [Gus00]

F. Gustafsson. Adaptive Filtering and Change Detection. Wiley, 2000. ISBN 0-471-49287-6.

[Hor00]

A. Horch. Condition Monitoring of Control Loops. Phd thesis TRITA-S3-REG0002, Department of Signals, Sensors and Systems, Royal Institute of Technology, Stockholm, Sweden, 2000.

[HR97]

A. W. Houghton and C. D. Reeve. Direction finding on spread spectrum signals using the time-domain filtered cross spectral density. IEE Proceedings - Radar, Sonar and Navigation, 144(6):315–320, December 1997.

[KG81]

H. Kurz and W. Goedecke. Digital parameter-adaptive control of processes with unknown dead time. Automatica, 17:245–252, January 1981.

[KQ92]

Simon Kingsley and Shaun Quegan. Understanding Radar Systems. McGrawHill, 1992. ISBN 0-07-707426-2.

[Lju99]

Lennart Ljung. System Identification: Theory for the User. Prentice-Hall, Upper Saddle River, N.J. USA, 2nd edition, 1999.

[Mat01]

The MathWorks, Inc. MATLAB Statistics Toolbox. User’s Guide. Version 3, 2001.

[Mon97] D. C. Montgomery. Design and Analysis of Experiments. Wiley, 1997. ISBN 0-471-15746-5. [Swa99]

Anthony Paul Swanda. PID Controller Performance Assessment Based on Closed-Loop Response Data. Phd thesis, University of California, Santa Barbara, California, USA, June 1999.

[Wik02] Maria Wikstr¨om. Utveckling och implementering av ett audiopejlsystem baserat p˚ a tidsdifferensm¨atning. Master’s thesis LiTH-ISY-EX-3277-2002, Link¨opings universitet, Link¨oping, Sweden, October 2002. In swedish.

1

2

A

Validation of confidence intervals

This appendix comments on the use and applicability of the confidence intervals in Section 4.2. The 4 RMS estimates (Chapter 3) have first gone through a transformation to make the variance more constant [Bj¨o03]. The transformation became (RMS error)ˆ(0.73028) (Figure A.1). Then ANOVA [Mon97] was executed and confidence intervals for pairwise comparisons [Mat01] plotted . The traditional ANOVA table is given in Table A.1. Since all p-values (the column Prob>F) are very small, all factors and interactions have effect with a very high confidence level. The confidence intervals are plotted in Section 4.2. For the ANOVA and confidence intervals to be valid some prerequisites must be fulfilled [Mon97, Bj¨o03]. These are usually tested by studying some validation graphs [Mon97, Bj¨o03] (Figures A.2-A.3) . We see in these graphs that the prerequisites are not completely fulfilled so we must be somewhat careful in the interpretation of the ANOVA and confidence interval. See [Mon97, Bj¨o03] for more information on this. Source Method InType SNR Sys Method*InType Method*SNR Method*Sys InType*SNR InType*Sys SNR*Sys Method*InType*SNR Method*InType*Sys Method*SNR*Sys InType*SNR*Sys Method*InType*SNR*Sys Error Total

Sum Sq. 50.4689 564.3458 1421.6913 527.049 21.1096 5.9101 48.857 21.597 13.6246 45.817 127.9938 11.3337 4.8604 327.4812 44.8829 2.5363 3239.5586

d.f. 3 2 1 3 6 3 9 2 6 3 6 18 9 6 18 288 383

Mean Sq. 16.823 282.1729 1421.6913 175.683 3.5183 1.97 5.4286 10.7985 2.2708 15.2723 21.3323 0.62965 0.54004 54.5802 2.4935 0.0088066

F 1910.2693 32041.1107 161434.944 19949.0399 399.5042 223.6997 616.42 1226.1859 257.8478 1734.1939 2422.3104 71.4977 61.3222 6197.6538 283.1397

Prob>F 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

Table A.1: Analysis of Variance table for all methods. Constrained (Type III) sums of squares. Positive transformation: (RMS error)ˆ(0.73028).

1

Before transformation.: Cell std vs. mean

0 −2

log10(cellStd(:))

−4 −6 −8 −10 −12 −14 −16 −16

−14

−12

−10

−8 −6 −4 log10(cellAbsMeans(:))

−2

0

2

Figure A.1: Plot (without transform) for choosing a variance-stabilizing transform [Mon97] for ANOVA. The transformation is chosen by fitting a straight line to the data points by the least squares method. The outlier in the lower left corner is ignored when calculating the transformation. It is due to zero variance (always the same time-delay estimate) for one factor level combination.

2

t149b5: Normal plot of residuals

Probability

20

0.50

15

0.25 0.10 0.05 0.02 0.01 0.003 0.001 −0.4

10 5

−0.2

0 Data

0.2

0 −0.4

0.4

t149b5: Residuals vs. time

0.6 0.4

0.4

0.2

0.2

0 −0.2

−0.2

0

0.2

0.4

0.6

t149b5: Residuals vs. fitted value

0.6

Residuals

Residuals

No resids=384

25

0.75

−0.4

t149b5: Histogram of residuals

30

0.999 0.997 0.99 0.98 0.95 0.90

0 −0.2

0

100

200 Time

300

−0.4

400

0

2

4 Fitted value

6

8

Figure A.2: Residual analysis for ANOVA and confidence intervals. Ideally the residuals should be Gaussian and the residuals vs. time and fitted value should be within a horizontal band and be structureless [Mon97, Bj¨o03].

3

t149b5: Residuals vs. Method

0.09

t149b5: Residuals vs. InType

0.1 0.09

0.08

0.08

0.07

0.07

0.06 Residuals

Residuals

0.06 0.05 0.04

0.05 0.04

0.03

0.03

0.02

0.02

0.01 0

0.01

1

1.5

2

2.5 Method

3

3.5

0

4

t149b5: Residuals vs. SNR

0.09

1

1.2

1.4

1.6

1.8

2 InType

2.2

2.4

2.6

2.8

3

t149b5: Residuals vs. Sys

0.12

0.08 0.1 0.07 0.08

0.05

Residuals

Residuals

0.06

0.04 0.03

0.06

0.04

0.02 0.02 0.01 0

1

1.2

1.4

1.6

1.8

0

2

SNR

1

1.5

2

2.5 Sys

3

3.5

4

Figure A.3: Standard deviation of residuals versus factor levels for ANOVA and confidence intervals. Ideally the standard deviation for all factor levels should be equal[Mon97, Bj¨o03].

4

Suggest Documents