© ISO 2011 – All rights reserved
ISO TC 69/SC 6 N Date: 2011-03-9
ISO/WD 15725-1 ISO TC 69/SC 6/WG 1 Secretariat:
Accuracy (trueness and precision) of measurement methods and results — Part 1: Introduction and basic principles Exactitude (justesse et fidélité) des résultats et méthodes de mesure — Partie 1: Introduction et principes de base
Warning This document is not an ISO International Standard. It is distributed for review and comment. It is subject to change without notice and may not be referred to as an International Standard. Recipients of this draft are invited to submit, with their comments, notification of any relevant patent rights of which they are aware and to provide supporting documentation.
Document type: International Standard Document subtype: Document stage: (20) Preparatory Document language: E
ISO/WD 15725-1
Copyright notice This ISO document is a working draft or committee draft and is copyright-protected by ISO. While the reproduction of working drafts or committee drafts in any form for use by participants in the ISO standards development process is permitted without prior permission from ISO, neither this document nor any extract from it may be reproduced, stored or transmitted in any form for any other purpose without prior written permission from ISO. Requests for permission to reproduce this document for the purpose of selling it should be addressed as shown below or to ISO's member body in the country of the requester: [Indicate the full address, telephone number, fax number, telex number, and electronic mail address, as appropriate, of the Copyright Manager of the ISO member body responsible for the secretariat of the TC or SC within the framework of which the working document has been prepared.] Reproduction for sales purposes may be subject to royalty payments or a licensing agreement. Violators may be prosecuted.
ii
© ISO 2011 – All rights reserved
ISO/WD 15725-1
Contents 1
Page
Scope ......................................................................................................................................................1
2
Normative references............................................................................................................................1
3
Terms and definitions ...........................................................................................................................2
4 4.1 4.2 4.2.1 4.2.2 4.2.3 4.2.4 4.2.5 4.3 4.4 4.4.1 4.4.2 4.4.3 4.4.4 4.5 4.5.1 4.5.2 4.6 4.7 4.7.1 4.7.2 4.7.3 4.7.4 4.8 4.8.1 4.8.2 4.8.3
Organization of collaborative studies .................................................................................................9 Collaborative study ...............................................................................................................................9 Personnel involved in collaborative study .........................................................................................9 Working group .......................................................................................................................................9 Statistical functions ..............................................................................................................................9 Executive functions.............................................................................................................................10 Supervisors..........................................................................................................................................10 Operators .............................................................................................................................................11 Planning ...............................................................................................................................................11 Conditions of collaborative study .....................................................................................................12 Measurement or test method .............................................................................................................12 Collaborative study item.....................................................................................................................12 Short intervals of time.........................................................................................................................13 Participating laboratories ...................................................................................................................16 Statistical analysis of a collaborative study .....................................................................................16 Preliminary considerations ................................................................................................................16 Tabulation of the results and notation used.....................................................................................17 Selection of laboratories for the accuracy experiment ...................................................................18 Utilization of trueness and precision ................................................................................................18 Checking the acceptability of test results ........................................................................................18 Stability of test results within a laboratory.......................................................................................18 Assessing the performance of a laboratory .....................................................................................18 Comparing alternative measurement methods................................................................................18 The report to, and the decisions to be taken by, the panel.............................................................18 The report by the statistical expert....................................................................................................18 Decisions to be taken by the panel ...................................................................................................18 Full report.............................................................................................................................................19
5
Flowchart..............................................................................................................................................19
6 6.1 6.1.1 6.1.2 6.1.3 6.1.4 6.2
Statistical model ..................................................................................................................................22 Principles of the basic model.............................................................................................................22 General mean.......................................................................................................................................23 Laboratory effect .................................................................................................................................23 Random error term ..............................................................................................................................23 Bias of the measurement method......................................................................................................24 Alternative models ..............................................................................................................................24
Annex A (informative) Symbols and abbreviation used in ISO 15725_Part 1 ...........................................25 Annex B (informative) Homogeneity and stability checks of samples......................................................26
© ISO 2011 – All rights reserved
iii
ISO/WD 15725-1
Foreword ISO (the International Organization for Standardization) is a worldwide federation of national standards bodies (ISO member bodies). The work of preparing International Standards is normally carried out through ISO technical committees. Each member body interested in a subject for which a technical committee has been established has the right to be represented on that committee. International organizations, governmental and non-governmental, in liaison with ISO, also take part in the work. ISO collaborates closely with the International Electrotechnical Commission (IEC) on all matters of electrotechnical standardization. International Standards are drafted in accordance with the rules given in the ISO/IEC Directives, Part 2. The main task of technical committees is to prepare International Standards. Draft International Standards adopted by the technical committees are circulated to the member bodies for voting. Publication as an International Standard requires approval by at least 75 % of the member bodies casting a vote. Attention is drawn to the possibility that some of the elements of this document may be the subject of patent rights. ISO shall not be held responsible for identifying any or all such patent rights. ISO 15725-1 was prepared by Technical Committee ISO/TC 69, Applications of statistical methods, Subcommittee SC 6, Measurement methods and results. This first edition of ISO 15725-1 cancels and replaces ISO 5725-1:1994, part of ISO 5725-2:1994. It also incorporates the Technical Corrigendum ISO 5725-1:1994/Cor. 1:1998. ISO 15725 consists of the following parts, under the general title Accuracy (trueness and precision) of measurement methods and results: ⎯ Part 1: Introduction and basic principles ⎯ Part 2: Basic method to evaluate the accuracy of measurement methods and results ⎯ Part 3: Alternative methods to evaluate the accuracy of measurement methods and results ⎯ Part 4: Practical (other) uses of accuracy values
iv
© ISO 2011 – All rights reserved
ISO/WD 15725-1
Introduction Standardization of the performance of a measurement method [definition 3.1] has existed for many years [ISO 5725]. The release of each part of this Standard is an opportunity to improve the applicable methods of data analysis that have existed for many years in ISO 5725. ISO 15725, like ISO 5725, may be applied to a wide range of measurands. It is concerned with those users of a measurement method who need criteria to assess the performance of the method. ISO 15725 presents a method of analysis of inter-laboratory comparisons that can be adapted to intralaboratory comparisons. Users’ criteria, concerning operator, equipment, etc. for the tests can be different. In particular, intra-laboratory comparison is realised under specific laboratory conditions. The main changes in progressing from ISO 5725 to this Standard are: •
Provision of a description of improved statistical techniques for data analysis,
•
Presentation of a flowchart for software developers, and data sets to benchmark the compliance of an implementation of these techniques,
•
Presentation of information at two levels: the procedure given in the body of the Standard; and in an appendix, demonstration of the underlying concepts including bibliographic references,
•
Emphasizing the role of the coordinator or organiser: e.g. in using knowledge of limits and justifying appropriate statistical techniques, and
•
Presentation of examples of practical exploitation of criteria (relating to detection of outliers, etc.) to support decisions made by the user and to make links with other relevant standards.
The objective of the ISO 15725 series is a)
to provide useful definitions,
b)
to provide procedures to assess accuracy (trueness and precision) of measurement methods and results;
c)
to give guidance and examples to use in practice of trueness and precision data
Precision and trueness data obtained from collaborative studies are important characteristics of measurement and test methods. Precision and trueness data •
support standardisation of test methods, and are mandatory for proposed ISO standard test methods;
•
support regulatory agencies in selecting methods;
•
assist laboratories in assessing uncertainty of results obtained from standard methods (this topic is covered in detail in ISO 21748);
•
allow the comparison of the performance of different methods.
To facilitate its use, ISO 15725 is structured in four parts referenced 1, 2, 3 and 4 (as opposed to six parts of ISO 5725); parts 2 and 3 can be implemented independently of one another. But parts 2 and 3 rely on definitions and concepts presented in part 1:
© ISO 2011 – All rights reserved
v
ISO/WD 15725-1
Part 1: General principles and definitions Part 2: Basic method to evaluate the accuracy of measurement methods and results Part 3: Alternative methods to evaluate the accuracy of measurement methods and results Part 4: Practical uses of accuracy values The method of measurement is the result of a process that may include several steps such as those described in the Figure 1. The ISO 15725 series propose to evaluate the performance of a measurement method by evaluating its accuracy (trueness and precision of measurement method).
Development or Modification of measurement method to answer to needs Process of measurement under control : Control Chart
Implementation of measurement method
Interlaboratory study : Accuracy of measurement method ISO 5725 Series
Optimization of measurement method : Experimental design
The measurement method is defined: Writing procedure
Intralaboratory study : Characterization of measurement method
Figure 1 — Life-cycle of a measurement method The objective of ISO 15725 – Part 1 is: a)
to state general principles and give definitions for the ISO 15725 series;
b)
to describe conditions, constraints and resources necessary to characterize a measurement method or a result, and
c)
to define an organisational scheme according to required objectives and economical, organisational and technical constraints.
The objective of ISO 15725 – Part 2 is:
vi
a)
to develop a basic method (based on a statistical model) to assess the accuracy of a measurement method or results;
b)
to define the necessary and sufficient conditions to apply the basic method;
© ISO 2011 – All rights reserved
ISO/WD 15725-1
c)
to describe the statistical tests to identify outlier results of participants;
d)
to provide the tools to estimate: 1. Precision (evaluated by repeatability standard deviation, intermediate precision standard deviations, reproducibility standard deviation), and 2. Trueness (evaluated by the bias of the measurement method).The objective of ISO 15725 – Part 3 is to provide alternatives to the basic method, in particular the use of robust statistics as an alternative to rejection of outliers.
The objective of ISO 15725 – Part 4 is to give guidance for the use of the values obtained in Parts 2 or 3, for instance:
NOTE
-
the use repeatability, intermediate precision and reproducibility limits for decision support;
-
the use accuracy and precision for method validation;
-
to compare with values obtained by different methods;
-
to compare with reference values;
-
to evaluate uncertainty of measurement; Various examples are given in the appendices (with algorithms and software) of Parts 2, 3 and 4.
© ISO 2011 – All rights reserved
vii
WORKING DRAFT
ISO/WD 15725-1
Accuracy (trueness and precision) of measurement methods and results — Part 1: Introduction and basic principles
1
Scope
ISO 15725 Part 1 defines an organisational scheme for acquisition of trueness and precision data by interlaboratory study, and includes the necessary terminology, basic model and principles. ISO 15725 Part 1 is concerned with the evaluation of the accuracy of a single defined measurement procedure, for example a proposed standard method, by means of inter-laboratory comparison, in which essentially identical items or a single stable item are provided to participants. ISO 15725 Part 1 is concerned exclusively with measurement procedures that yield measured values on a continuous scale and give a single numerical value as the test or measurement result, although this single value may be the outcome of a calculation from a set of observations. ISO 15725 Part 1 is applicable to a very wide range of materials, manufactured or naturally occurring, provided that due consideration is given to any heterogeneity of the material. This Standard is not applicable to proficiency testing or Reference Material Certification. ISO 15725 Part 1 does not include methods of calculation. NOTE 1 For the purposes of this Standard, quantitative test methods are considered to be measurement methods, and all references to measurement methods are to be understood as applying equally to quantitative test methods. NOTE 2 The principles of ISO 15725 apply to discontinuous quantities where the quantity is measured on an interval scale and the minimum scale interval is small compared to the dispersion of results. This is typical for count data when the counts are large and the measurement variance is much larger than one.
2
Normative references
The following referenced documents are indispensable for the application of this document. For dated references, only the edition cited applies. For undated references, the latest edition of the referenced document (including any amendments) applies. ISO 3534-2:2006, Statistics — Vocabulary and symbols — Part 2: Applied statistics ISO 3534-3:2006, Statistics — Vocabulary and symbols — Part 3: Design of experiments ISO/IEC Guide 99:2007, International vocabulary of metrology — Basic and general concepts and associated terms (VIM) IEC 60050-300:2001, International Electrotechnical Vocabulary - Electrical and electronic measurements and measuring instruments ISO/TS 21748 2004 Guidance for the use of repeatability, reproducibility and trueness estimates in measurement uncertainty estimation
© ISO 2011 – All rights reserved
1
ISO/WD 15725-1
3
Terms and definitions
For the purposes of this document, the following terms and definitions apply. The symbols used in ISO 15725 are given in Annex A. The following correspondence must be considered when using ISO 15725 series: ⎯ ‘Measurement method’ used in the title is to be understood as a ‘measurement procedure’ as defined in ISO/IEC Guide 99:2007. The measurement procedure may include and define steps of sampling, sample preparation, conditioning or data treatment. In this case, the performance of the ‘measurement method’ consists on the evaluation of the performance of a combination of several methods (preparation, testing, and treatment). [See chapter ”Organization of collaborative studies”]. ⎯ When applied to a test method or a measurement method, ‘accuracy’ refers to the accuracy of the results obtained with the method, ⎯ The definition of ‘measurement result’ is that from ISO/IEC Guide 99:2007, but it restricted to the case where the result is expressed as a single value, that may be the outcome of a calculation from a set of observations, ⎯ ‘The true value’ is in practice never known. In general, an accepted reference value is used. 3.1 measurement procedure detailed description of a measurement according to one or more measurement principles and to a given measurement method, based on a measurement model and including any calculation to obtain a measurement result NOTE 1 A measurement procedure is usually documented in sufficient detail to enable an operator to perform a measurement. NOTE 2
A measurement procedure can include a statement concerning a target measurement uncertainty.
NOTE 3
A measurement procedure is sometimes called a standard operating procedure, abbreviated SOP.
[ISO/IEC Guide 99:2007, definition 2.6] NOTE 4
Normally, a measurement procedure is the implementation of the measurement method.
3.2 measurand quantity intended to be measured NOTE 1 The specification of a measurand requires knowledge of the kind of quantity, description of the state of the phenomenon, body, or substance carrying the quantity, including any relevant component, and the chemical entities involved. NOTE 2 In second edition of the VIM and in IEC 60050-300:2001, the measurand is defined as the ‘quantity subject to measurement’. NOTE 3 The measurement, including the measuring system and the conditions under which the measurement is carried out, might change the phenomenon, body, or substance such that the quantity being measured may differ from the measurand as defined. In this case, adequate correction is necessary.
2
© ISO 2011 – All rights reserved
ISO/WD 15725-1
EXAMPLE 1 The potential difference between the terminals of a battery may decrease when using a voltmeter with a significant internal conductance to perform the measurement. The open-circuit potential difference can be calculated from the internal resistances of the battery and the voltmeter. EXAMPLE 2 The length of a steel rod in equilibrium with the ambient Celsius temperature of 23 °C will be different from the length at the specified temperature of 20 °C, which is the measurand. In this case, a correction is necessary. NOTE 4 In chemistry “analyte”, or the name of a substance or compound, are terms sometimes used for “measurand”. This usage is erroneous because these terms do not refer to quantities.
[ISO/IEC Guide 99:2007, definition 2.3] NOTE 5 Regarding NOTE 4, a correct usage would be, for example, “the molar concentration of the analyte” or “the mass fraction of analyte present or extracted”.
3.3 measurement result set of quantity values being attributed to a measurand together with any other available relevant information NOTE 1 A measurement result generally contains “relevant information” about the set of quantity values, such that some may be more representative of the measurand than others. This may be expressed in the form of a probability density function (PDF). NOTE 2 A measurement result is generally expressed as a single measured quantity value and a measurement uncertainty. If the measurement uncertainty is considered to be negligible for some purpose, the measurement result may be expressed as a single measured quantity value. In many fields, this is the common way of expressing a measurement result. NOTE 3 In the traditional literature and in the previous edition of the VIM, measurement result was defined as a value attributed to a measurand and explained to mean an indication, or an uncorrected result, or a corrected result, according to the context. [ISO/IEC Guide 99:2007, definition 2.9] NOTE 4
For the purposes of this standard, a measurement result assumed to be expressed as a single value.
3.4 outlier value in a set of values that is inconsistent with other values in the set NOTE Statistical tests and the significance level to be used to identify outliers in trueness and precision experiments are defined in Part 2.
3.5 true value value which characterizes a quantity or quantitative characteristic perfectly defined in the conditions which exist when that quantity or quantitative characteristic considered
Mis en forme : Definition, Sans numérotation ni puces
NOTE The true value of a quantity or quantitative characteristic is a theoretical concept and, in general cannot be known exactly
Mis en forme : Note, Sans numérotation ni puces
[ISO 3534-2:2006, clause 3.2.5] 3.6 accepted reference value value that serves as an agreed-upon reference for comparison NOTE a)
The accepted reference value is derived as: a theoretical or established value, based on scientific principles;
© ISO 2011 – All rights reserved
3
ISO/WD 15725-1
b)
an assigned or certified value, based on experimental work of some national or international organisation;
c)
a consensus or certified value, based on collaborative experimental work under the auspices of a scientific or technical group;
d)
the expectation, i.e. the mean of a specified set of measurements, when a), b) and c) are not available.
[ISO 3534-2:2006, clause 3.2.7] 3.7 accuracy closeness of agreement between a test result or measurement result and the true value NOTE 1
In practice, the accepted reference value is substituted for the true value.
NOTE 2 The term “accuracy”, when applied to a set of test or measurement results, involves a combination of random components and a common systematic error or bias component. NOTE 3
Accuracy refers to a combination of trueness and precision.
[ISO 3534-2:2006, clause 3.3.1] 3.8 trueness closeness of agreement between the expectation of a test result or a measurement result and a true value NOTE 1
The measurement of trueness is usually expressed in terms of bias.
NOTE 2
Trueness is sometimes referred to as “accuracy of the mean”. This usage is not recommended.
NOTE 3
In practice, the accepted reference value is substituted for the true value.
[ISO 3534-2:2006, clause 3.3.3] NOTE 4 The ‘expectation of a test result’ is to be interpreted as the expectation of the random variable of which the test result is an instance.
3.9 bias difference between the expectation of a test result or measurement result and a true value NOTE 1 Bias is the total systematic error as contrasted to random error. There may be one or more systematic error components contributing to the bias. A larger systematic difference from the true value is reflected by a larger bias value. NOTE 2 The bias of a measuring instrument is normally estimated by averaging the error over an appropriate number of repeated measurements. The error of indication is the “indication of a measuring instrument minus a true value of the corresponding input quantity”. NOTE 3
In practice, the accepted reference value is substituted for the true value.
[ISO 3534-2:2006, clause 3.3.2] 3.10 bias of the measurement method method bias difference between the expectation of a test result or measurement result using that method and a true value NOTE 1 value.
4
The true value is in practice never known. In practice, the accepted reference value is substituted for the true
© ISO 2011 – All rights reserved
ISO/WD 15725-1
NOTE 2
The bias of the measurement method may depend on the test item.
3.11 laboratory effect effect attributed to a laboratories’ factor included as a factor in a statistical model , “L” in the statistical model 3.12 factor predictor variable that is varied with the intent of assessing its effect on the response variable NOTE 1
A factor may provide an assignable cause for the outcome of an experiment.
NOTE 2
The use of factor is more specific than its generic use as a synonym for predictor variable
NOTE 3
A factor may be associated with the creation of block
[ISO 3534-3:1999, clause 1.5]
3.13 precision closeness of agreement between independent test/measurement results obtained under stipulated conditions NOTE 1 Precision depends only on the distribution of random errors and does not relate to the true value or the specified value. NOTE 2 The measure of precision is usually expressed in terms of imprecision and computed as a standard deviation of the test results or measurement results. Less precision is reflected by a larger standard deviation. NOTE 3 Quantitative measures of precision depend critically on the stipulated conditions. Repeatability conditions and reproducibility conditions are particular sets of extreme stipulated conditions.
[ISO 3534-2:2006, clause 3.3.4] NOTE 4 Precision depends only on the distribution of random errors and does not relate to the true value or the accepted value.
3.14 repeatability precision under repeatability conditions [ISO 3534-2:2006, clause 3.3.5] NOTE
Repeatability can be expressed quantitatively in terms of the dispersion characteristics of the results.
3.15 repeatability conditions observation conditions where independent test/measurement results are obtained with the same method on identical test/measurement items in the same test or measuring facility by the same operator using the same equipment within short intervals of time NOTE
Repeatability conditions include:
⎯
the same measurement procedure;
⎯
the same operator;
⎯
the same measuring or test equipment used under the same conditions;
© ISO 2011 – All rights reserved
5
ISO/WD 15725-1
⎯
the same location;
⎯
repetition over a short period of time.
[ISO 3534-2:2006, clause 3.3.6] 3.16 repeatability standard deviation standard deviation of test results or measurement results obtained under repeatability conditions NOTE 1 It is a measure of the dispersion of the distribution of test or measurement results under repeatability conditions. NOTE 2 Similarly, “repeatability variance” and “repeatability coefficient of variation” can be defined and used as measures of the dispersion of test or measurement results under repeatability conditions.
[ISO 3534-2:2006, clause 3.3.7] NOTE 3 Knowledge is needed of the probability distribution from which the value is regarded as drawn; often it can be taken as normal.
3.17 repeatability critical difference value less than or equal to which the absolute difference between two final values, each of them representing a series of test results or measurement results obtained under repeatability conditions, is expected to be with a specified probability EXAMPLE
Examples of final results are the mean and the median of the series of results; the series itself may consist of only one result.
[ISO 3534-2:2006, clause 3.3.8] NOTE The calculation of a repeatability critical difference usually requires a known or assumed distribution for differences.
3.18 repeatability limit repeatability critical difference for a specified probability of 95 % [ISO 3534-2:2006, clause 3.3.9] NOTE 1
The calculation of a critical difference usually requires a known or assumed distribution for differences.
NOTE 2
The symbol used for repeatability limit is r.
3.19 reproducibility precision under reproducibility conditions NOTE 1
Reproducibility can be expressed quantitatively in terms of the dispersion characteristics of the results.
NOTE 2
Results are usually understood to be corrected results.
[ISO 3534-2:2006, clause 3.3.10] 3.20 reproducibility conditions observation conditions where independent test/measurement results are obtained with the same method on identical test/measurement items in different test or measurement facilities with different operators using different equipment
6
© ISO 2011 – All rights reserved
ISO/WD 15725-1
[ISO 3534-2:2006, clause 3.3.11] 3.21 reproducibility standard deviation standard deviation of test results or measurement results obtained under reproducibility conditions NOTE 1 It is a measure of the dispersion of the distribution of test or measurement results under reproducibility conditions. NOTE 2 Similarly, “reproducibility variance” and “reproducibility coefficient of variation” can be defined and used as measures of the dispersion of test or measurement results under reproducibility conditions.
[ISO 3534-2:2006, clause 3.3.12] NOTE 3
Reproducibility standard deviation is always greater than or equal to the repeatability standard deviation.
3.22 reproducibility critical difference value less than or equal to which the absolute difference between two final values, each of them representing a series of test results or measurement results obtained under reproducibility conditions, is expected to be with a specified probability EXAMPLE Instances of final results are the mean and the median of the series of results; the series itself may consist of only one result.
[ISO 3534-2:2006, clause 3.3.13] NOTE
The calculation of a critical difference usually requires a known or assumed distribution for differences.
3.23 reproducibility limit reproducibility critical difference for a specified probability of 95 % [ISO 3534-2:2006, clause 3.3.14] NOTE 1
The calculation of a critical difference usually requires a known or assumed distribution for differences
NOTE 2
The symbol used for reproducibility limit is R.
3.24 intermediate precision precision under intermediate precision conditions [ISO 3534-2:2006, clause 3.3.15] 3.25 intermediate precision conditions conditions where test results or measurement results are obtained with the same method, on identical test/measurement items in the same test or measurement facility, under some different operating condition NOTE 1
There are four elements to the operating condition: time, calibration, operator and equipment.
NOTE 2
A test house is an example of a test facility. A metrology laboratory is an example of a measurement facility.
[ISO 3534-2:2006, clause 3.3.16] NOTE 3 It is the clear identification of the particular conditions above that are varied during the course of a study that give a particular set of “intermediate precision conditions”: ⎯
time,
© ISO 2011 – All rights reserved
7
ISO/WD 15725-1
⎯
equipment,
⎯
operator,
⎯
calibration,
⎯
temperature,
⎯
pressure, …
NOTE 4
The above conditions may change independently.
3.26 intermediate precision standard deviation standard deviation of test results or measurement results obtained under intermediate precision conditions [ISO 3534-2:2006, clause 3.3.17] 3.27 intermediate precision critical difference value less than or equal to which the absolute difference between two final values, each of them representing a series of test results or measurement results obtained under intermediate precision conditions, is expected to be with a specified probability [ISO 3534-2:2006, clause 3.3.18] NOTE
The calculation of a critical difference usually requires a known or assumed distribution for differences.
3.28 intermediate precision limit intermediate precision critical difference for a specified probability of 95 % [ISO 3534-2:2006, clause 3.3.19] NOTE
The calculation of a critical difference usually requires a known or assumed distribution for differences.
3.29 level or measuring interval set of values of quantities of the same kind that can be measured by a given measuring instrument or measuring system with specified instrumental uncertainty, under defined conditions NOTE 1 In some fields, the term is “measuring range” or “measurement range”. NOTE 2 The lower limit of a measuring interval should not be confused with detection limit
[ISO/IEC Guide 99:2007, definition 4.7] NOTE The level can be taken as the general average of the test or measurement results from all laboratories for the collaborative study item.
3.30 collaborative study interlaboratory experiment in which the performance of each laboratory is assessed using the same standard measurement method on identical material 3.31 collaborative study item sample, product, artefact, reference material, piece of equipment, measurement standard, data set or other information used for collaborative study
8
© ISO 2011 – All rights reserved
ISO/WD 15725-1
4
Organization of collaborative studies
4.1 Collaborative study The accuracy (trueness and precision) measurement should be determined from a series of test or measurement results reported by laboratories participating in an interlaboratory study organized for that purpose and under the supervision of the panel (defined in part 2). Such an interlaboratory study is called a "collaborative study". The estimates of accuracy (precision and trueness) derived from such a study should always be quoted as being valid only for tests carried out according to the defined measurement method. Before deciding the validity of a particular method, there is benefit in developing, optimizing and characterizing that method. The aim of this standard proposes ways to achieve this characterisation (see Figure 1).
4.2 Personnel involved in collaborative study 4.2.1
Working group
This working group should be persons familiar with the measurement method or test method accurately characterize and a person with knowledge of the statistical analysis and interpretation. The tasks of the working group are: a)
to plan and coordinate the experiment;
b)
to decide on number of laboratories, levels and measurements to be made, and the number of significant figures to be required;
c)
to appoint someone for the statistical functions (see 4.2.2);
d)
to appoint someone for executive functions (see 4.2.3);
e)
to consider the instructions to be issued to the laboratory supervisors in addition to the measurement method;
f)
to decide whether some operators may be allowed to carry out a few unofficial measurements in order to regain experience of the method after a long interval (such measurements shall never be carried out on the official collaborative samples);
g)
to choose a study item that is stable and homogeneous
h)
to discuss the report of the statistical analysis on completion of the analysis of the test results;
i)
to establish final values for the repeatability standard deviation and the reproducibility standard deviation;
j)
to decide if further actions are required to improve the standard for the measurement method or with the regard top laboratories whose results have been rejected as outliers.
4.2.2
Statistical functions
At least one member of the working group has experience in statistical design and analysis of experiments. His/her tasks are: a)
to contribute his/her specialized knowledge in designing the collaborative study;
b)
to analyse the data;
c)
to write a report for submission to the panel following the instructions contained in 4.8.
© ISO 2011 – All rights reserved
9
ISO/WD 15725-1
4.2.3
Executive functions
The actual organization of the experiment should be entrusted to a single laboratory. A member of the staff of that laboratory should take full responsibility; he/she is called the executive officer and is appointed by the panel. The tasks of the executive officer are: a)
to enlist the cooperation of the requisite number of laboratories and to ensure that supervisors are appointed;
b)
to organize and supervise the preparation of the materials and samples, to evaluate the homogeneity and stability and the dispatch of the samples; for each level, an adequate quantity of material should be set aside as a reserve stock;
c)
to draft instructions of collaborative study covering all the important points and circulate them to the supervisors early enough in advance for them to raise any comments or queries and to ensure that operators selected are those who would normally carry out such measurements in routine operations;
d)
to design suitable forms to the operator to use as a working record and for the supervisor to report the test results to the requisite number of significant figures (such forms may include the name of the operator, the dates on which samples were received and measured, the equipment used and any other relevant information) or any calculation;
e)
to deal with any queries from laboratories regarding the performance of measurements;
f)
to see that an overall time schedule is maintained;
g)
to collect the data forms and present them to the statistical expert.
4.2.4
Supervisors
A staff member in each of the participating laboratories should be made responsible for organizing the actual performance of the measurements, in keeping with instructions received from the executive officer, and for reporting test results. The tasks of the supervisor are: a)
ensure that the operators selected are those who would normally carry out such measurements in routine operations;
b)
to hand out the samples to the operator(s) in keeping with the instructions of the executive officer (and to provide material for familiarization experiments, if necessary);
c)
to supervise the execution of the measurements (the supervisor shall not take part in performing the measurements);
d)
to ensure that the operators carry out the required number of measurements;
e)
to ensure adherence to the set timetable for performing the measurements;
f)
to collect the test results recorded to the agreed number of decimal places, including any anomalies and difficulties experienced, and comments made by the operators.
The supervisor of each laboratory should write a full report which should contain the following information:
10
a)
the test results, entered legibly by their originator on the forms provided, not transcribed or typed (computer or testing machine printout may be acceptable as an alternative);
b)
the original observed values or readings (if any) from which the test results were derived, entered legibly by the operator on the forms provided, not transcribed or typed;
© ISO 2011 – All rights reserved
ISO/WD 15725-1
c)
comments by the operators on the standard for the measurement method;
d)
information about irregularities or disturbances that may have occurred during the measurements, including any change of operator that may have occurred, together with a statement as to which measurements were performed by which operator, and the reasons for any missing results;
e)
the date(s) on which the samples were received;
f)
the date(s) on which each sample was measured;
g)
information about the equipment used, if relevant;
h)
any other relevant information.
4.2.5
Operators
In each laboratory the measurements shall be carried out by one operator selected as being representative of those likely to perform the measurements in normal operations. Because the object of the experiment is to determine the precision obtainable by the general population of operators working from the standard measurement method, in general the operators should not be given amplifications to the standard for the measurement method. However, it should be pointed out to the operators that the purpose of the exercise is to discover the extent to which results can vary in practice, so that there will be less temptation for them to discard or rework results that they feel are inconsistent. Although normally the operators should receive no supplementary amplifications to the standard measurement method, they should be encouraged to comment on the standard and, in particular, to state whether the instructions contained in it are sufficiently unambiguous and clear. The tasks of the operators are: a)
to perform the measurements according to the standard measurement method;
b)
to report any anomalies or difficulties experienced; it is better to report a mistake than to adjust the test results because one or two missing test results will not spoil the experiment and many indicate a deficiency in the standard;
c)
to comment on the adequacy of the instructions in the standard; operators should report any occasions when they are unable to follow their instructions as this may also indicate a deficiency in the standard.
4.3
Planning
Before initiating a collaborative study, it is appropriate to evaluate the feasibility of the study by asking the key questions: a)
Is a satisfactory defined method available for the measurement method?
b)
Is there a list of influent factors concerning the method?
c)
Is there a need to draft a protocol to explain hard points to the participants?
d)
How many laboratories should be recruited to cooperate in the collaborative study?
e)
How should the laboratories be recruited, and what requirements should they satisfy?
f)
What is the range of levels encountered in practice?
© ISO 2011 – All rights reserved
11
ISO/WD 15725-1
g)
How many levels should be used in the experiment?
h)
What are suitable materials to represent these levels and how should they be prepared? (Identical, same batch, homogeneous, stable?)
i)
What number of replicates should be specified?
j)
What time-frame should be specified for the completion of all measurements?
k)
Are any special precautions needed to ensure that identical materials are measured in the same state in all laboratories? Choice of circulation scheme, packaging, shipment, transportation, storage…
l)
Is the basic model appropriate, or should a modified one be considered?
4.4 4.4.1
Conditions of collaborative study Measurement or test method
To avoid any discrepancies and to reduce dispersion, all participants must have the same detailed measurement or test method describing how the measurement shall be carried out, and how the measurement item should be obtained and prepared. The measurement method under investigation should be one that has been optimized. Such a method should be robust, i.e., small variations in the procedure should not produce unexpectedly large changes in the results (compared with cycle life). The document specifying the measurement method should be unambiguous and complete. All essential operations concerning the environment for the procedure, the reagents and apparatus, metrological traceability of equipment, and the preparation of the test specimen should be included in the measurement method. In order to analyze the measurement method or test method to establish the set of influencing factors, it is pertinent to prepare an Ishikawa diagram as in Figure 2.
Personnel
Environment
Method
result Handling
Traceability
Sampling
Equipment Means
Items
Figure 2 — Ishikawa diagram It is important to indicate how the results should be calculated and reported, and the number of significant decimal digits to be provided. The organizers will send participants a template (in Microsoft Excel, for example) to enter their results. 4.4.2
Collaborative study item
In a collaborative study, samples of a specific material or specimens of a specific product are sent from a central point to a number of laboratories in different locations, different countries, or even in different continents. The definition of repeatability conditions (3.14), stating that the measurements in these laboratories shall be performed on identical test items, refers to the moment when these measurements are
12
© ISO 2011 – All rights reserved
ISO/WD 15725-1
actually carried out. To achieve this, two conditions have to be satisfied: a)
homogeneity: the samples should be identical or at least from the same batch when dispatched to the laboratories;
b)
stability: the samples should remain stable during transport and during the different time intervals that may elapse before the measurements are actually performed.
In organizing a collaborative study, both stability and homogeneity conditions should be carefully observed. Tests for homogeneity and stability The procedures describing the verification of homogeneity and stability check are presented in Annex B (Homogeneity and Stability checks of samples). The item to be used in a collaborative study to determine the accuracy of measurement method laboratories should represent fully those to which the measurement method is expected to be applied in normal use and if possible metrological traceability. If the item is heterogeneous the effect of heterogeneity should be included in the accuracy values. When measurements are to be performed on solid materials that cannot be homogenized (such as metals, rubber or textile fabrics) and when the measurements cannot be repeated on the same test piece, inhomogeneity in the test material will form an essential component of the precision of the measurement and the assumption of identical material no longer holds. Precision experiments can still be carried out, but the values of precision may only be valid for the particular material used and should be quoted as such. A more universal use of precision as determined would be acceptable only if it can be demonstrated that the values do not differ appreciably between materials produced at different times or by different suppliers. A more elaborate experiment would be required in such a case, which is beyond the scope of this International Standard. In general, where destructive testing is involved, the contribution to the variability in the test results arising from differences between the specimens on which the measurements are performed should either be negligible compared to the variability of the measurement method itself, or else should form an inherent part of the variability of the measurement method, and thus be a component of precision included in repeatability. When the materials under measurement might change with time, it might be appropriate to specify the times at which the samples are to be measured (example solution of nitrate with validity of 15 days). Some test items are not transportable, such as an oil storage tank. In such cases, measurement by different laboratories means that different operators are sent with their measuring equipment to the test site. In other cases, the quantity being measured may be transitory or variable, such as water flow in a river, when care should be taken that the different measurements are made, as closely as possible, under the same conditions. The guiding principle should always be that the objective is to determine the ability to repeat the same measurement. For the assessment of trueness, at least one of the materials used should have an accepted reference value. If it is likely that trueness varies with level, materials with accepted reference values will be needed at several levels. 4.4.3
Short intervals of time
According to the definition of repeatability conditions (3.14), measured values used for the determination of repeatability should be made under constant operating conditions; i.e., during the time period in which measurement is made, relevant factors should not be varying. In particular, the equipment should not be recalibrated during this period unless this is an essential part of every single measurement. In practice, tests under repeatability conditions should be conducted in as short a time as possible in order to minimize changes in those factors, such as environmental, which cannot be guaranteed constant.
© ISO 2011 – All rights reserved
13
ISO/WD 15725-1
There is also a second consideration that may affect the time interval between measurements, and that is that the test results are assumed to be independently obtained. If it is feared that previous test results may influence subsequent results (and so reduce the estimate of repeatability variance), it may be necessary to provide separate specimens coded in such a way that an operator will not know which are supposedly identical. Instructions would be given as to the order in which those specimens are to be measured, and presumably that order would be randomized so that the "identical" items are not measured together. Hence the time interval between repeated measurements might appear to conflict with the need to complete all measurement within a short time period. Common sense should prevail.
14
© ISO 2011 – All rights reserved
ISO/WD 15725-1
Types of collaborative studies: Sequential: Distribute test item to first participant. After measurement, participant returns item to the reference laboratory or sends it to the next participant for measurement or back to the reference laboratory to check the stability. The stability checks should be made regularly (see Figure 3). Lab 3 Lab 1 Lab 2 Lab 4 Lab 5
Lab p
R e fe re n c e L a b o ra to ry
Lab 6
Lab p – 1 Lab 7 Lab 8
… …
Figure 3 Sequential distribution When the measurements have to be performed on a discrete item (such as a car or temperature sensor) that is not altered by measurement, they could, in principle at least, be carried out by different laboratories (sequential circuit). There would be a consequent risk of loss or damage during transport, particularly if there would be many participating laboratories possibly in different countries. If different items are to be used in different laboratories, then they shall be selected in such a way as to ensure that they can be presumed to be identical for purposes of the test. Simultaneous: Distribute test items, selected randomly from a batch of items, individually to all participants for measurement. After measurement, participants return items to the reference laboratory (see Figure 4).
Batch
Item1
Lab 1
Item 2
Lab 2
Item3
Lab 3
………
………
…….. Item p
Lab p
Figure 4 Simultaneous distribution
© ISO 2011 – All rights reserved
15
Commentaire [MGC1] :
ISO/WD 15725-1 Commentaire [MGC2] : Already said.
4.4.4
Participating laboratories
Representativeness of the participating laboratories From a statistical point of view, those laboratories participating in any experiment to estimate accuracy should have been chosen at random from all laboratories that use the measurement method. Volunteers might not constitute a reasonable representation. However, other practical considerations, such as the requirement that participating laboratories not be from one sector or be only large laboratories, can improve representativeness. A basic assumption underlying part 1 of IS0 15725 is that, for a standard measurement method, repeatability will be, at least approximately, the same for all laboratories applying the standard procedure, so that it is permissible to establish one common average repeatability standard deviation that will be applicable to any laboratory. However, any laboratory can, by carrying out a series of measurements under repeatability conditions, produce an estimate of its own repeatability standard deviation for the measurement method and compare it with the common standard value. The participating laboratories should not consist exclusively of those that have gained special experience during the process of standardizing the method. Neither should they consist of specialist "reference" laboratories in order to demonstrate the accuracy to which the method can perform in expert hands. Number of participating laboratories and laboratory measurement results There should be a sufficient number of laboratories as well as measurement results for each participant to assess performance parameters with confidence. The number of laboratories to be recruited to participate in a collaborative study and the number of test results required from each laboratory at each level of the test are interdependent. In Part 2 of this Standard a method is proposed to calculate the number of laboratories and number of repetitions per participant.
4.5 Statistical analysis of a collaborative study 4.5.1
Preliminary considerations
The analysis of the data, which should be considered as a statistical problem to be solved by a statistical expert, involves three successive stages: a)
critical examination of the data in order to identify and treat outliers or other irregularities and to test the suitability of the model;
b)
computation of preliminary values of precision trueness and means for each level separately;
c)
establishment of final values of precision and means, including the establishment of a relationship between precision and the level m when the analysis indicates that such a relationship may exist.
The analysis first computes, for each level separately, estimates of ⎯ the repeatability variance
σ r2 ,
⎯ the between-laboratory variance ⎯ the reproducibility variance ⎯ the mean m ,
16
sr2 σ L2 ,
σ R2 = σ L2 + σ r2 ,
sL2 sR2 = sL2 + sr2 ˆ m
© ISO 2011 – All rights reserved
ISO/WD 15725-1
⎯
the intermediate precision if it is necessary to consider more factors than the factor laboratory in the model
The analysis includes a systematic application of statistical test for outliers, a great variety of are available from the literature and which could be used for the purposes of this ISO 15725 series. For practical reasons, only a limited number of these tests have been incorporated in ISO 15725 Part 2. 4.5.2
Tabulation of the results and notation used
Cells Each combination of a laboratory and a level is called a cell of the precision experiment. In the ideal case, the results of an experiment with p laboratories and 4 levels consist of a table with pq cells, each containing 1replicate test results that can all be used for computing the repeatability standard deviation and the reproducibility standard deviation. This ideal situation is not, however, always attained in practice. Departures occur owing to redundant data, missing data and outliers. Redundant data Sometimes a laboratory may carry out and report more than the number test results officially specified. In that case, the supervisor shall report why this was done and which are the correct test results. If the answer is that they are all equally valid, then a random selection should be made from those available test results to choose the planned number of test results for analysis. Missing data In other cases, some of the test results may be missing, for example because of loss of a sample or a mistake in performing the measurement. The analysis recommended is such that completely empty cells can simply be ignored, while partly empty cells can be taken into account by the standard computational procedure. Outliers These are entries among the original test results, or in the tables derived from them, that deviate so much When several unexplained abnormal test results occur at different levels within the same laboratory, then that laboratory may be considered to be an outlier, having too high a within-laboratory variance and/or too large a systematic error in the level of its test results. It may then be reasonable to discard some or all of the data from such an outlying laboratory. This part of ISO 15725 does not provide a statistical test by which suspected laboratories may be judged (see ISO 15725-2). The primary decision should be the responsibility of the statistical expert, but all rejected laboratories shall be reported to the panel for further action. Outlying laboratories When several unexplained abnormal test results occur at different levels within the same laboratory, then that laboratory may be considered to be an outlier, having too high a within-laboratory variance and/or too large a systematic error in the level of its test results. It may then be reasonable to discard some or all of the data from such an outlying laboratory. ISO 15725 series do not provide a statistical test by which suspected laboratories may be judged. The primary decision should be the responsibility of the statistical expert, but all rejected laboratories shall be reported to the panel for further action. Erroneous data Obviously erroneous data should be investigated and corrected or discarded.
© ISO 2011 – All rights reserved
17
ISO/WD 15725-1
4.6 Selection of laboratories for the accuracy experiment From a statistical point of view, those laboratories participating in any experiment to estimate accuracy should have been chosen from all the laboratories using the same measurement method
4.7 Utilization of trueness and precision Practical applications of trueness and precision values are covered in detail in ISO 15725-4. Some examples are as follows. 4.7.1
Checking the acceptability of test results
A product specification could require repeated measurement to be obtained under repeatability conditions. A repeatability standard deviation may be used in these circumstances to check the acceptability of the test results and to decide what action should be taken if they are not acceptable. When both a supplier and a purchaser measure the same material and their results differ, repeatability and reproducibility standard deviations may be used to decide if the difference is of a size that is to be expected with the measurement method. 4.7.2
Stability of test results within a laboratory
By carrying out regular measurements on reference materials, a laboratory can check the stability of its results and produce evidence to demonstrate its competence with respect to both the bias and the repeatability of its testing. 4.7.3
Assessing the performance of a laboratory
Laboratory accreditation schemes are becoming increasingly widespread. Knowledge of the trueness and precision of a measurement method allows the bias and repeatability of a candidate laboratory to be assessed, either using reference materials or an interlaboratory experiment. 4.7.4
Comparing alternative measurement methods
Two measurement methods may be available for measuring the same property, one being simpler and less expensive than the other but less generally applicable. Trueness and precision values may be used to justify the use of the less expensive method for some restricted range of materials.
4.8 The report to, and the decisions to be taken by, the panel 4.8.1
The report by the statistical expert
Having completed the statistical analysis, the statistical expert should write a report to be submitted to the panel. In this report the following information should be given: a)
a full account of the observations received from the operators and/or supervisors concerning the standard for the measurement method;
b)
a full account of the laboratories that have been rejected as outlying laboratories together with the reasons for their rejection;
c)
a full account of any stragglers and/or statistical outliers that were discovered, and whether these were explained and corrected, or discarded.;
4.8.2
Decisions to be taken by the panel
The panel should then discuss this report and take decisions concerning the following questions.
18
© ISO 2011 – All rights reserved
ISO/WD 15725-1
a)
Are the discordant results, stragglers or outliers, if any, due to defects in the description of the standard for the measurement method?
b)
What action should be taken with respect to rejected outlying laboratories?
c)
Do the results of the outlying laboratories and/or the comments received from the operators and supervisors indicate the need to improve the standard for the measurement method? If so, what are the improvements required?
d)
Do the results of the precision experiment justify the establishment of values of the repeatability standard deviation and reproducibility standard deviation? If so, what are those values, in what form should they be published, and what is the region in which the precision data apply?
4.8.3
Full report
A report setting out the reasons for the work and how it was organized, including the report by the statistician and setting out agreed conclusions, should be prepared by the executive officer for approval by the panel. Some graphical presentation of consistency or variability is often useful. The report should be circulated to those responsible for authorizing the work and to other interested parties.
5
Flowchart
This flowchart is a presentation of the general organisation of a collaborative study, and provides a link between the different parts of ISO 15725 series. Economical considerations are not included but are considered in each process. The following steps are managed by the working group, as defined in 4.2.1.
© ISO 2011 – All rights reserved
19
ISO/WD 15725-1
ISO 15725 Part 1
Definition of organisation scheme - Working group, objectives Æ see § 4.2 - Description of the quantitative measurement method Æ see § 4.4.1 - Participants: minimum number required Æ see § 4.4.4 - Choice of the comparison item (Standard, CRM, sample, …) Æ see § 4.4.2
CRM and standard compulsory for any trueness evaluation
Feasibility of collaborative study - Study of stability and homogeneity of comparison item Æ see Annex B - Definition of logistic requirements (packaging, storage, transportation) Æ see § 4.4.3 => Choice of circulation scheme: simultaneous or round robin test
Choice of results exploitation method - BASIC when: - the comparison item is homogeneous and stable (the standard deviation of inter-sample ≤ 0, 3 × standard deviation of reproducibility of the method expected), - the number of data is sufficient (indicative data in case of the laboratory effect: number of laboratories (p ≥ 8) and number of replicates (n ≥ 2) => p*n ≥ 16),
- ALTERNATIVE when the basic method is not appropriate and/or: - the measurement accuracy is evaluated for approximation, or - the measurement on one sample influences the result of a subsequent measurement on another sample of the same material.
Basic method is detailed in ISO 15725 Part 2 Alternative method is detailed in ISO 15725 Part 3
Preliminary collaborative study with few labs Documents of collaborative study (for each participant) - Presentation: objectives, comparison item, coordination team - Comparison principle - Measurement method: detailed description of the method concerned by the collaborative study - Results form to be filled and transmitted (significant number, any other information…) - Timetable (sample reception date, testing date, sample sending date…) no Is the number of effective participants superior or equal to required ? yes
Launch of collaborative study sending of samples
Management of timetable, data collection, circulation, information…
20
© ISO 2011 – All rights reserved
ISO/WD 15725-1
BASIC METHOD ISO 15725 Part 2
Alternative METHOD Preliminary technical review
ISO 15725 Part 3
Identification of erroneous data: determination of root causes for each atypical data and help to correction to ensure process control
ISO 15725 Part 1
Preliminary statistical review Identification of outlier data Æ see § 4.5
Statistical analysis of results
Statistical analysis of results
- Means Precision: - Repeatability variance - Intermediate precision variance(s), if appropriate - Reproducibility variance Trueness: - Bias of the measurement method
- Robust Means Precision: - Repeatability variance - Reproducibility variance Trueness: - Bias of the measurement method
Trueness & Precision of measurement method
Practical uses of collaborative study - Uncertainty of measurement method, - Comparison to a reference value, - Comparison of two methods, - Use of repeatability/reproducibility limits
ISO 15725 Part 4
Synthesis report Æ see § 4.8
ISO 15725 Part 1
- Campaign description : objectives, processing - Raw data, outlier data - Results of statistical analysis and conclusion - Document references
Synthesis report sent to each participants
Not compulsory
Figure 5 Flowchart
© ISO 2011 – All rights reserved
21
ISO/WD 15725-1
6
Statistical model
6.1 Principles of the basic model The measurement method is not modelled by a physical or chemical equation, but by a statistical model. This model is used as a basis to assess trueness and precision of the measurement method from the measurement results of the participating laboratories. This basic statistical model is formally a random effects model with the factor “laboratory” regarded as a set of random effects (one for each participating laboratory) and consequently not fixed. These effects are considered as random variables that are mutually independent and identically distributed (and therefore not fixed and unknown constants). Each individual laboratory is regarded as randomly drawn from a larger population of laboratories. This population is considered to be characterized by a random variable with unknown expectation m and unknown variance σ L It is absolutely necessary to deal with a random effects model to consider the concepts of repeatability and reproducibility. It is also from this model that the variances of repeatability and reproducibility will be estimated. 2
The model is described as follows:
y i , j = m + Li + ε i , j y i , j jth observation of ith laboratory m general mean of the test
Li ith laboratory effect
ε i , j random error on each observation performed in repeatability conditions The ε i , j are regarded as mutually independent random variables with expectation zero and unknown variance
σ r2 , called the repeatability variance.
The Li are regarded as mutually independent random variables with
expectation zero and unknown variance independent of the
ε i,j .
σ L2
, called the laboratory variance; the Li are regarded as
The use of the above model confirms these variances and gives the covariances With the consequences of the above assumptions it is possible to write
E (ε i , j ) = 0 ∀ i ,
var( ε i , j ) = σ r2 , cov( ε i , j , Li ) = 0 ∀ i ≠ j .
E( Li ) = 0 ∀ i ,
var( Li ) = σ L2 ,
In summary: cov( y i , j , y i' , j'
cov( Li , Li' ) = 0 ∀ i ≠ i' .
⎧σ L2 + σ r2 for i = i ' and j = j ' ⎫ ⎪ ⎪ ) = ⎨σ L2 for i = i ' et j ≠ j ' ⎬ ⎪0 if not ⎪ ⎩ ⎭
The model is described in Figure 6
22
© ISO 2011 – All rights reserved
ISO/WD 15725-1
σ
Variation ofLaboratoryeffect L
L
Lab1
Lab2.
...
..
Labj
First step
i
σ
r
Variation of test results within a laboratory
Lab mean
Secondstep step
i
...
ε i ,j
y
y
i
i ,j
L +ε i
= m+ L + ε i
i ,j
step Third step
i ,j
m y i ,j General mean Figure 1
Figure 6 The expectations of
ε i , j and
Li are supposed to be equal to zero; actually, the laboratories Li belong to a
population of laboratories L of which the mean is equal to zero. This standard proposes to assess the variances. The variance of
ε is the within-laboratory variance σ r2 , and the variance of L is the between-
laboratories variance, σ . 2 L
The sum of between-laboratories variance and within-laboratory variance is the reproducibility variance, σ R , 2
that is σ = σ + σ , and makes it possible to determine the combined variance, taking into account all influence factors. This variance is called reproducibility variance. 2 R
6.1.1
2 L
2 r
General mean
The quantity m will be estimated from the data with the model, and is not necessary equal to the true value µ . In many technical situations, the test should be performed at different values of the general mean, called levels. 6.1.2
Laboratory effect
The laboratory effect is considered to be constant during any series of tests performed under repeatability conditions, but to differ in value for tests carried out under other conditions. With that model, repeatability conditions mean that the laboratory effect is constant. The procedures given in ISO 15725-2 are developed assuming that the distribution of laboratory effect is approximately normal, but in practice they apply to most distributions provided that they are unimodal. 6.1.3
Random error term
This term represents a random error occurring in every test result and the procedures given throughout this part of ISO 15725 are developed assuming that the distribution of this random error variable is approximately normal, but in practice they apply to most distributions provided that they are unimodal.
© ISO 2011 – All rights reserved
23
ISO/WD 15725-1
6.1.4
Bias of the measurement method
ˆ − µ where m ˆ is an estimate of the The bias δ of the measurement method may be estimated from δˆ = m general mean and µ is the true value (if known) or the accepted reference value.
6.2 Alternative models Extensions to the basic model are used when appropriate and are described in the relevant parts of ISO 15725
24
© ISO 2011 – All rights reserved
ISO/WD 15725-1
Annex A (informative) Symbols and abbreviation used in ISO 15725_Part 1
r
repeatability limit
R
reproducibility limit
m
general mean of the test
ˆ m
estimate of m
Li
ith laboratory effect
yi,j
jth observation of ith laboratory
ε i , j random error related to observation yi,j performed in repeatability conditions E(X) expectation of random variable X Var(X) variance of random variable X Cov(X,Y) covariance between random variables X and Y
σ r2 : repeatability variance, residual variance σˆ r2 = s r2 estimate of the repeatability variance, residual variance σ L2 between-laboratory variance σˆ L2 = s L2 estimate of between-laboratory variance σ R2 reproducibility variance
σˆ R2 = s R2 estimate of reproducibility variance µ true value or accepted reference value δ bias of the measurement method δˆ estimate of the bias
© ISO 2011 – All rights reserved
25
ISO/WD 15725-1
Annex B (informative) Homogeneity and stability checks of samples
Homogeneity: The test is held to confirm the homogeneity of each sample is as follows: The standard deviation of inter-sample ≤ 0, 3 × standard deviation of reproducibility of the method expected.
ss ≤ 0,3 sR The factor of 0.3 variation within samples fixed at most 10% of the standard deviation of reproducibility of the measurement method. If the criterion is not verified, the organizing team will have to choose another sample or estimate this variability to assess the method. NOTE:
Another criterion may be chosen with appropriate technical justification.
Stability: The test is held to validate the stability of each sample is as follows: It is to compare the difference ∆x in the average homogeneity values at the beginning and end of a specified time interval with the expected reproducibility standard deviation sR of the method: The absolute difference in average study of homogeneity in the overall average results of the study homogeneity ≤ 0.3 × standard deviation of reproducibility of the method expected.
∆x ≤ 0.3 s R If the criterion is not verified, the organizing team will assess possibilities of improvement (storage, preparation,).
26
© ISO 2011 – All rights reserved
Proposal structure for the revision ISO 5725 – 2 (future 15725-2) Red: modification the text in the current ISO 5725 Blue: new redaction Basic method to evaluate the accuracy (trueness and precision) of measurement method.
0
Introduction (trueness: information on reference material (traceability) / precision) 0.1 0.2 0.3
As in current standard As in current standard As in current standard with an introduction of trueness and removing intermediate precision As current standard and introducing intermediate precision Add eventually something coming from Part 3 and Part 4 of ISO 5725
0.4
1
Scope 1.1 ♦ ♦ ♦ ♦ 1.2
This part of ISO 15725 Amplifies the general …. Precision and trueness Basic method (Precision and trueness) Conception and design of the collaborative study (collaborative study) Method to evaluate Intermediate precision This part of ISO 15725 Continuous scale
1.3 Needs modifications. Deal with balanced and unbalanced data 1.4 1.5 1.6
Pool 1.4 and 1.5 Delete (not necessary)
2
Normative reference
3
Definitions
All definitions are in the ISO 15725 part 1. Specifics definitions for this part 2
4 Key points of the collaborative study This paragraph describe elementary points necessary to 4.1 4.2 4.3
Balanced and unbalanced data Constraints on the heterogeneity of the materials Estimates of the parameter in statistical model
Interim meeting TC69/SC6/WG1 –20-21 january 2011
1/3
5
Interlaboratory comparison Design 5.1
Item of comparison
5.2
Number of laboratory and number of repetition
5.3
Personnel involved
5.4
Analysis of measurement method
5.5
Planning and logistic
5.6
Collect and Data table Erroneous data (eliminate erroneous data) Table A original data Table B means Table C standard deviation
6
Statistical analysis of laboratory results 6.1
Scrutiny of results for consistency and outliers Redundant data Outliers Graphical technique? (to check consistency of data): Box plot; Youden plot;… Numerical technique (Cochran Grubbs plus other tests, new standard for outlier’s detection ISO 16269-4; Oct 2010)
6.2
Estimates of the statistical parameters
6.2.1 From the basic model Estimates of
m ,σ R2 ,σ r2
Analysis of variance‘s table for estimation 6.2.2 Deducted parameters Trueness Limit of repeatability Limit of reproducibility Critical difference of repeatability Critical difference of reproducibility 6.3
Establishing functional relationship between precisions values and the mean level
7
Open question Interlaboratory comparison Design with several factors within the laboratories (other part ?: Part 4 – Intermediate precision to evaluate accuracy (precision and trueness)new structure Æ15727-series 5 parts?) (Interlaboratory comparison Design to evaluate intermediate precision) Follow for evaluated intermediate precision the same protocol as for basic method except some particular points described below
Interim meeting TC69/SC6/WG1 –20-21 january 2011
2/3
7.1
Factor’s selection
7.2
Number of repetition by factor
7.3
Scrutiny of results for consistency and outliers
7.4
Estimates of the statistical parameters
Annex A : Algorithm Annex B Examples Annex C : ANOVA
Interim meeting TC69/SC6/WG1 –20-21 january 2011
3/3
