Uncertainty in Measurements – Estimation and Practical Use 4/5/2012 Speaker Anders Kallner, MD, PhD, Associate Professor of Clinical Chemistry, Karolinska Hospital, Stockholm, Sweden Anders Kallner, MD, PhD studied general chemistry at the University of Stockholm and organic chemistry at the International Union of Pure and Applied Chemistry (IUPAC) Elections Royal Institute of Technology in Stockholm before graduating in biochemistry (with a PhD) from the Karolinska Institute in 1967. He later earned his MD at the same university and became associate professor of Clinical Chemistry at the Karolinska Institute. He has held positions in county, regional, and university hospitals. Although he retired from Karolinska University Hospital in 2005, Dr. Kallner retains professional assignments in the laboratory and international organizations. He has given more than 250 invited lectures and has contributed to more than 180 publications. Dr. Kallner has held numerous memberships and leadership roles on numerous international committees, including the International Organization for Standardization (ISO), the International Federation of Clinical Chemistry and Laboratory Medicine (IFCC), and the IUPAC. Dr. Kallner has participated in the development of several CLSI Evaluation Protocols, and is currently the chairholder of the Subcommittee on Expression of Measurement Uncertainty in Laboratory Medicine (C51) and an active member of the CLSI Area Committee on Evaluation Protocols. Objectives At the conclusion of this program, participants will be able to:  Explain the concept of uncertainty  Estimate the uncertainty in measurements  Assess the Importance of bias in measurements Continuing Education Credit The Association of Public Health Laboratories (APHL) is approved as a provider of continuing education programs in the clinical laboratory sciences by the ASCLS P.A.C.E.® Program. Participants who successfully complete this program will be awarded 1 contact hour of continuing education credit. Florida CEU credit will be offered based on 1 contact hour. Continuing education credits are available to individuals who successfully complete the program and evaluation by Date. The evaluation password is 606op. Detailed directions for completing the evaluation and printing your certificate are on www.aphl.org/courses/pages/webinarevaluation.aspx. Archived Program The archived streaming video will be available within two weeks. Anyone from your site can view the Web archived program and/or complete the evaluation and print the CEU certificate for free. To register for the archive program go to http://www.aphl.org/courses/Pages/590-606-12.aspx and use the complementary discount code 606op in the discount box during registration. Comments, opinions, and evaluations expressed in this program do not constitute endorsement by APHL or CLSI. The APHL and CLSI do not authorize any program faculty to express personal opinion or evaluation as the position of APHL or CSLI. The use of trade names and commercial sources is for identification only and does not imply endorsement by the program sponsors. © This program is copyright protected by the speaker, CLSI and APHL. The material is to be used for this APHL program only. It is strictly forbidden to record the program or use any part of the material without permission from the author or APHL. Any unauthorized use of the written material or broadcasting, public performance, copying or re-recording constitutes an infringement of copyright laws.

 

4/2/2012

Measurement Uncertainty

Anders Kallner, MD, PhD Chairholder CLSI Document C51 Karolinska Hospital Stockholm, Sweden

1

Normative Reference

Joint Committee for Guides in Metrology (JCGM) BIPM IEC IFCC ILAC ISO IUPAC IUPAP OIML

2

Measurement Uncertainty

Evaluation of measurement data – Guide to the expression of uncertainty in measurement (GUM) JCGM 100: 2008 www.bipm.org (International Bureau of Weights and Measures) Quantifying Uncertainty in Analytical Measurement, 2nd ed., 2000 QUAM 2000.1 www.eurachem.org/index.php/publications/guides

3

1

4/2/2012

Measurement Uncertainty

CLSI document C51: Expression of Measurement Uncertainty in Laboratory Medicine; Approved Guideline This document describes a practical approach to develop and calculate useful estimates of measurement uncertainty.

4

Vocabulary

JCGM 200:2008

International vocabulary of metrology — Basic and general concepts and associated terms (VIM) www.bipm.org

5

What Do We Measure?

quantity property of a phenomenon, body, or substance, where the property has a magnitude that can be expressed as a number and a reference NOTE: The preferred IUPAC-IFCC format for designations of quantities in laboratory y medicine is “System—Component; y p kind-of-quantity,” q y S-Cholesterol; mass concentration kind of quantity aspect common to mutually comparable quantities NOTE: The quantities diameter, circumference, and wavelength are quantities of the same kind, namely of the kind of quantity called length.

6

2

4/2/2012

What Do We Measure?

The kind of quantity defines the dimension of the result and thus the unit. In clinical chemistry we usually work with: Mass concentration --- g/L Substance concentration --- mol/L Activity concentration --- katal/L (U/L) Number concentration --- 103/L / A basic principle of the International System (SI) is that there must be no prefix in the denumerator; thus µg/L is advised instead of ng/Ml and g/L rather than mg/mL. The numerical values remain the same. In clinical chemistry, ”substance concentration” is preferred particularly when there are several molecular species with the same active epitope, eg, Vitamin D: 25(OH)D = 25(OH)D2 + 25(OH)D3, improve!

7

Measurand

measurand quantity intended to be measured NOTE: The specification of a measurand requires knowledge of the kind of quantity, and the chemical entities involved.

8

Interval and Range

Interval includes the begining and end, whereas range is the difference between the end points of the interval. Thus, an interval is defined by two numbers, and range is defined by only one. Example: Reference interval: 138-142 mmol/L Measuring interval: 130-150 mmol/L Measuring range: 20 mmol/L

9

3

4/2/2012

Performance Characteristics

accuracy of measurement, accuracy closeness of agreement between a measured quantity value and a true quantity value of a measurand

trueness closeness of agreement between the average of an infinite number of replicate measured quantity values and a reference quantity value

10

Performance Characteristics

Accuracy describes closeness between observation and true value. Trueness describes closeness between mean and true value.

Accuracy and trueness are concepts. Corresponding metrics are: Accuracy – inaccuracy Trueness – bias

11

Performance Characteristics

precision closeness of agreement between indications or measured quantity values obtained by replicate measurements on the same or similar objects under specified conditions Precision is a concept and cannot be measured. Its metric is imprecision. imprecision Imprecision Standard deviation (s(X)) Coefficient of variation (%CV(X)) Repeatability: only time differs between measurements Reproducibility: allows any changes in the method but refers to the same quantity

12

4

4/2/2012

Performance Characteristics

sensitivity quotient of the change in an indication of a measuring system and the corresponding change in a value of a quantity being measured detection limit, limit of detection measured quantity value, obtained by a given measurement procedure, for which the probability of falsely claiming the absence of a component in a material is β, given a probability α of falsely claiming its presence

13

Measurement Uncertainty

uncertainty of measurement/uncertainty non-negative parameter characterizing the dispersion of the quantity values being attributed to a measurand, based on the information used The non-negative parameter is the standard uncertainty = standard deviation Dispersion of results around a best estimate, with a given probability

14

“Error

is an idealized concept and errors cannot be known exactly.”

Systematic error = Bias Random error = Imprecision Total error = Imprecision + Bias =Uncertainty

Total error: TE=z x s + b Rilibäk: ∆=sqrt(k2 x s2 + b2) Maximum allowable deviation Bias: b=mean - ref. (”true”) value Uncertainty: Best estimate and interval of confidence defined by coverage factor, k.

15

5

4/2/2012

Performance Concepts and Quantities Performance Characteristics, Concepts

T Trueness

Accuracy

Precision

16

Modified from Menditto et al. Accred Qual Assur (2007)

Performance Concepts and Quantities

Type of Errors, Quantity

Performance Characteristics, Concepts

Systematic y Error

Trueness

Total Error

Accuracy

Random Error

Precision

17

Modified from Menditto et al. Accred Qual Assur (2007)

Performance Concepts and Quantities Performance Characteristics, Concepts

Performance Characteristics, Quantity

Systematic y error

Trueness

Bias

Total error

Accuracy

Measurement Uncertainty

Random error

Precision

Imprecision

Type of Errors, Quantity

18

Modified from Menditto et al. Accred Qual Assur (2007)

6

4/2/2012

Ishikawa, Fish-Bone diagram

The basic format of this figure is variously termed "Ishikawa diagram," "cause-and-effect diagram," and "fish-bone diagram." 19

Uncertainty Budget

Input Quantity Sample indication (measurement signal) Calibrator indication (measurement signal)

Value

Standard Uncertainty

Relative Uncertainty

Ss

u(Ss)

u(Ss)/|Ss|

Scal

u(Scal)

u(Scal)/|Scal|

Blank indication (measurement signal)

S0

u(S0)

u(S0)/|S0|

Calibrator concentration

ccal

u(ccal)

u(ccal)/|ccal|

Dilution factor

d

u(d)

u(d)/|d|

Matrix effect (eg, interferences)

Em

u(Em)

u(Em)/|Em|

Nonspecified effects

Eu

u(Eu)

u(Eu)/|Eu|

20

Propagation of Uncertainties

Sums and differences

A  B u  A u B  u  A  B   u  A  u  B  2

2

Products and ratios

A  B u  A  B   u  A   u B        A  B  A   B  2

2

21

7

4/2/2012

Propagation of Uncertainties Spreadsheet Solution

Kragten, EURACHEM (EQUAM)

Cl 

U  Vol  U  Crea P  Crea  time

22

Example 1

Example 1: Addition of two volumes A and B with uncertainties ML81 u(A) and u(B). Estimate the combined uncertainty u(A+B) of the volume C. Assume A = 5 mL, u(A) = 0.1 mL; B = 95 mL, u(B) = 1.0 mL  u( A )  0.1

u( B )  1.0 u  A  B   0.12  12  1.01  1.005 mL The uncertainty of the total volume is less than the sum of the separate volumes.

23

Example 2

Reference Change Value

Example 2: S-Cholesterol concentration is measured once on two different instruments, one with an imprecision of 0.1 mmol/L, the other with 0.5 mmol/L. The first result was 5.6 mmol/L, the second 4.7 mmol/L. Is the difference significant at a level of confidence of 95%? A  B ; u  A   0. 2; u  B   0 .5 u  A  B   u  A  u B  2

2

Diff  k  u  A  B   2  0.12  0.5 2  2  0.51  1.0 mmol / L

Because the difference is only 0.9 mmol/L the difference is not significant on the desired level of confidence. 24

8

Slide 23 ML81

Inset 9-pt space on either side of plus sign. Megan Larrisey, 3/21/2012

4/2/2012

Example 3

Minimal Significant Difference

Example 3: Suppose in Example 2 two samples were measured by the same equipment and thus the same uncertainty (mean of the two: 0.3 mmol/L). Is the difference 0.9 mmol/L significant at a level of confidence of 95%? A B

u  A  B   u  A  u  B  ; 2

2

u  A  u B ; u  A  B   u  A 2 Diff  k  u  A  B   2  u  A 2  2  0.42  0.8 mmol / L Because the difference is 0.9 mmol/L the difference is significant on the desired level of confidence. Rule of thumb: u(A) x 3 25

Standard Uncertainty Type A and Type B

Calculation (estimation) of standard uncertainty Type A: Repeated measurements, calculate the standard deviation = standard uncertaintyy u(X)=S(X) For example, see CLSI documents EP05 and EP15, and internal quality control data. Type B: Estimate from experience, rectangular and triangular distributions

26

Rectangular Distribution

For example a volume flask contains more than 95 mL but less than 105 mL. Estimate the uncertainty of the nominal volume of 100 mL!

a

mL

s( A )  

a  2.9 mL 3 27

9

4/2/2012

Triangular Distibution

For example a volume flask contains more than 95 mL but less than 105 mL. Estimate the uncertainty of the nominal volume of 100 mL! a

mL

s( A )  

a  2.0 mL 6 A triangular distribution may be skewed 28

Uncertainty Budget

Standard Relative Input Quantity Value Uncertainty Uncertainty Sample indication Ss u(Ss) u(Ss)/|Ss| (measurement signal) Calibrator indication Scal u(Scal) u(Scal)/|Scal| (measurement signal) Blank indication S0 u(S0) u(S0)/|S0| (measurement signal) u(ccal) u(ccal)/|ccal| Calibrator concentration ccal Dilution factor d u(d) u(d)/|d| Matrix effect u(Em) u(Em)/|Em| Em (eg, interferences) u(Eu) u(Eu)/|Eu| Nons-pecified effects Eu

Estimate Type* Type A Type A Type B

Source Replication experiment Replication experiment Previous study

Type B Manufacturer Type B Professional judgment Type B

Literature

Type B Professional judgment

29

Tools in Search of Root Cause

Input Quantity Sample indication (measurement signal) Calibrator indication (measurement signal) Blank indication (measurement signal) Calibrator concentration

Value

Standard Relative Uncertaint Uncertaint Estimat y y e Type*

Ss

u(Ss)

Scal

u(Scal)

u(Ss)/|Ss|

Type A

u(Scal)/|Scal| Type A

S0

u(S0)

ccal

u(ccal)

d

u(d)

Matrix effect (eg, interferences)

Em

u(Em)

u(Em)/|Em| Type B

Nonspecified effects

Eu

u(Eu)

u(Eu)/|Eu|

Dilution factor

u(S0)/|S0|

Type B

u(ccal)/|ccal| Type B u(d)/|d|

Type B

Type B

Source Replication experiment Replication experiment Previous study Manufacturer Professional judgment Literature Professional judgment

30

10

4/2/2012

Summary; Standard Uncertainty Standard uncertainty u(xi) equals standard deviation SD(xi) u(xi) combined according to the measuring function or propagation rules is the combined uncertainty uc(X).

uc  X   u x1   u  x2   ...  u  xn  2

2

2

The combined uncertainty multiplied by a coverage factor (k) is the expanded uncertainty U(X). The coverage factor must always be stated, eg, 2 for a confidence level of about 95%.

U  X   k  uc  X ; k  2 

31

Bias

“Error is an idealized concept and errors cannot be known exactly”. ’True value’ is not known Bias = ’Observed value’ minus ’true value’ Comparing with reference material Comparing with results of reference method Comparing with results of previous laboratory method The bias can be estimated with an uncertainty. The bias can be compensated, reduced, eliminated, leaving and addition to the combined uncertainty

32

Elimination of Bias

33

11

4/2/2012

Elimination of Bias

u2comb=u2CRM+u2Obs +u2recal

34

Recalibration Strategy

When a bias is found to be small relative to the uncertainty of the measurement, it is not necessary to correct. Any bias correction that is clinically insignificant adds little or no value. If a bias is significant relative to the uncertainty of the measurement or to clinical utility, it may indicate that the measurement system is out of calibration or is otherwise producing invalid results and corrective actions are required. Any modification of a measurement system’s standard calibration protocol needs to be fully documented and verified.

35

Define measurand (5.0) Identify input quantities,  establish uncertainty  budget (6.1)

Bottom‐up (6) Top‐down (7)  App B

Create uncertainty budget  (6.2 Uncertainty by Type B  (6.3 App A) Uncertainty by Type A  ((7.3))

Uncertainty by Type A  (6)

Combine by measurement  function (6.4)

Estimate ubias  correction (8) Combine with other  identifed uncertainties  (6.1 6.5) Review and verify model  (7.5‐7.6) 36

12

4/2/2012

Summary Estimate the standard uncertainty by: Type A procedures, ie, measurements and standard statistics Type B procedures, estimation, eg, assuming rectangular or the bias:triangular distributions difference between measured values and a reference; if significant relative to the uncertainty or to clinical utility, recalibrate and estimate the uncertainty of the recalibration. Use propagation rules to estimate a combined uncertainty. Assign a coverage factor appropriate for the desired level of confidence. The uncertainty approach will move the responsibility for unbiased results from the end user to the producer - the laboratory!

37

13