ISO

Getting Practical about Measurement Uncertainty

 

James O. Westgard

 

 

The 2012 revision of ISO 15189 includes some significant changes regarding the laboratory’s responsibilities for uncertainty of measurement (MU).  The guidance in the previous edition [1] was that “the laboratory should determine the uncertainty of results, where relevant and possible.”  The phrase “where relevant and possible” allowed considerable room for argument, first (where relevant) whether physicians really want this information and know what to do with it, and second, (where possible) whether the GUM methodology for determining uncertainty makes it too complicated and impractical to estimate MU in a medical laboratory.  Because of this clause, there was considerable debate about whether laboratories should try to estimate MU and whether there were any practical methodologies for doing so. 

 

New ISO 15189 guidance (2012)

 

Section 5.5.1.4 in the 2012 guideline [2] is significantly different:

 

“The laboratory shall determine measurement uncertainty for each measurement procedure in the examination phases used to report measured quantity values on patients’ samples.  The laboratory shall define the performance requirements for the measurement uncertainty of each measurement procedure and regularly review estimates of measurement uncertainty.”

 

Remember that in ISO-speak, the word shall means the laboratory must determine measurement uncertainty, not that the laboratory might do this where relevant and possible.  In addition, the laboratory shall define performance requirements for measurement uncertainty, i.e., define guidance on the need for laboratories to define goals for how good their tests should be, then evaluate how good they are by comparison to their defined goals.    

 

In addition, the notes provide new guidance on how measurement uncertainty can be determined:

 

“Note 1. The relevant uncertainty components are those associated with the actual measurement process, commencing with the presentation of the sample to the measurement procedure and ending with the output of the measured value.

 

“Note 2.  Measurement uncertainties may be calculated using quantity values obtained by the measurement of quality control materials under intermediate precision conditions that include as many routine changes as reasonable possible in the standard operation of a measurement procedure, e.g., changes of reagent and calibrator batches, different operators, scheduled instrument maintenance.

 

“Note 3.  Examples of the practical utility of measurement uncertainty estimates include confirmation that patients values meet quality goals set by the laboratory and meaningful comparison of a patient value with a previous value of the same type or with a clinical decision value.”

 

Note 1 clearly limits MU to the analytic phase of the total testing process.  Note 2 recommends estimation from routine SQC data over a period of time that includes common changes or process variables that would contribute to MU.  Note 3 again brings in the responsibility to define goals for how good a test should be in order to periodically evaluate estimates of MU.

 

Getting Practical

 

The practical estimation of MU comes down to calculating the SD from SQC data, then multiplying that SD by a factor of 2 to provide a conventional 95% confidence limit for a test result.  The SD is known as the standard measurement uncertainty, the factor of 2 is called the coverage factor, and the 95% limit or interval is known as the expanded measurement uncertainty.

 

There is no mention of bias in this guidance, not even a mention of the uncertainty in the estimate of bias, which has generally been included in previous recommendations for top-down estimates.  Thus, the new ISO 15189 guidance says (1) you must do this and (2) you can do this just with SQC data collected over some extended period of time, not just one month.

 

I asked Graham White about this, suspecting that he would know how the final language was arrived at and whether there was any hidden meaning.  He mentioned that the guidance does not say you can’t or shouldn’t include the uncertainty in the estimate of bias and he further stated that another ISO committee will be making more specific recommendations on estimation of MU.  My interpretation is that laboratories, in the beginning, will be able to employ a simple methodology that primarily makes use of mid-term SQC data, but will be expected over time to improve those estimates, following later guidance and more rigorous models.

 

Intermediate precision conditions

 

ISO 15189 does not specifically define “intermediate precision conditions”, but that term applies to something between “repeatability conditions” and “reproducibility conditions,” which are defined in ISO and CLSI guidelines as follows:

 

“Repeatability condition of measurement, repeatability condition – condition of measurement, out of a set of conditions that includes the same measurement procedure, same operators, same measuring system, same operating conditions and same location, and replicate measurements on the same or similar obnjects over a short period of time.  Note: In chemistry, the term “within-ruin” or “intraserial” or “intrarun precision condition of measurement” is sometimes used to designate this concept.”

 

“Reproducibility condition of measurement, reproducibility condition – condition of measurement, out of a set of conditions that includes different locations, operators, measurement systems, and replicate measurements on the same or similar objects.  Note 1: The different measuring systems may use different measurement procedures; Note 2. A specification should give the conditions changed and unchanged, to the extent practical; Note 3. In chemistry, the terms “between-laboratories,” “interlaboratory,” or “among-laboratories precision conditions of measurement” are sometimes used to designiate this concept.”

 

Intermediate conditions imply something between these two states, generally meaning within one lab, but with changes between reagent lots, calibrator lots, operators, operating conditions, routine maintenance, periodic service, etc.  The practical issue is the appropriate time period for collecting and analyzing SQC data.  This time period will depend on the particular operating conditions for an individual test or analyzer, such as how often runs are performed, how often operators change, how frequent maintenance is performed, how often there are changes in reagent and calibrator lots, etc.  In addition, other factors that should be considered include the the number of measurements needed to obtain a reliable estimate of an SD, the frequency of SQC, and the time period over which data is collected to establish control limits.

 

Number of measurements for reliable estimate of SD.   The reliability of an estimate of the SD can be characterized by the confidence limit for that estimate, which depends on the number of measurements collected.  While there is a “rule of thumb” that a minimum of 20 control measurements should be used to calculate an SD for setting control limits, many more are needed to obtain a reliable estimate of the SD.  For example, if we assume a true standard deviation of 10 units, the 90% confidence interval will range from 7.4 to 15.9 when N=20, i.e., an SD as low as 7.4 could be observed, which is 26% low, or an SD as high as 15.9 could be observed, which is 59% high.  For N=100, the confidence interval is 9.0 to 11.3, i.e., the reliability of the estimate of the SD is much better, within approximately 10% of the correct value.  Therefore, it would seem prudent to aim for at least 100 measurements when estimating MU.

 

SQC frequency.  There is no standard practice for SQC frequency, but many laboratories, globally and in the US, tend to follow the CLIA guideline that a minimum of 2 levels of controls be analyzed per day.  Of course, high volume laboratories will often analyze many more controls per day.  On the other hand, US laboratories that have implemented Equivalent Quality Control (EQC) procedures may only be analyzing controls once per week or even once per month.  Likewise, the new emerging practice of Risk-based QC Plans may lead to low frequency of SQC, particularly in point-of-care applications.  Clearly the practicality of estimating MU from SQC data will depend on having a sufficient number of control measurements to provide a reliable estimate of the SD.  A reliable estimate of MU may not be obtainable for unit-use devices used in point-of-care applications, even though knowledge of the quality in these settings is critically important for patient treatment. 

 

            Cumulative control limits.  Given the difficulty of obtaining a reliable estimate of an SD, C24A3 [3] recommends that laboratories utilize several months data to establish cumulative control limits.  For example, if a laboratory analyzes 2 levels of controls per day, then data will be needed over 100 days to provide reliable SDs at the 2 levels.  This is the basis for the C24A3 recommendation that laboratories should combine control data from 6 consecutive monthly periods, calculate a cumulative SD, and implement control limits based on that cumulative SD. 

 

Summary

 

Measurement uncertainty is now a certainty!  The 2012 edition of ISO 15189 requires that medical laboratories determine measurement uncertainty.  The guideline also identifies a simple and practical methodology using SQC data obtained under “intermediate precision conditions, ”  i.e., a single laboratory and measurement principle, but with the changes in routine operating conditions (operations, reagent lots, calibrator lots, etc.).  The laboratory should calculate a mid-term SD and utilize this estimate to express the standard uncertainty, then multiply by a coverage factor of 2 to express an expanded measurement uncertainty (95% confidence limit or interval). 

 

While there is no specific guidance for how many control measurements are needed, the estimate of the SD will be more reliable if at least 100 data points are included, which will often require that SQC data be collected over a period of several months.  A period of 6 months should be practical in many laboratories and matches the CLSI recommendation for establishing control limits from a cumulative SD obtained from 6 successive months of routine SQC data.

 

None of this actually applies to US laboratories that operate under the CLIA regulations, rather than ISO 15189 accreditation.  Nonetheless, US laboratories should consider how to implement a methodology for determining MU because that will be part of the global standard of practice for quality management in medical laboratories.

 

 

References

 

1.      ISO 15189:2007.  Medical laboratories – Requirements for quality and competence.  2^nd  ed.  International Organization for Standards, Geneva, Switzerland, 2007.

2.      ISO 15189:2012.  Medical laboratories – Requirements for quality and competence.  3^rd ed.  International Organization for Standards, Geneva, Switzerland, 2012.

3.      CLSI C24A3.  Statistical Quality Control for Quantitative Measurement Procedures: Principles and Definitions. Clinical and Laboratory Standards Institute, Wayne, PA, 2006.

 

See also other discussions of MU on this website

 

·         Westgard JO. Update on Measurement Uncertainty: New CLSI C51A Guidance.  February 2012.  www.wetgard.com/clsi-c51.htm

·         White G.  The Hitchhiker’s Guide to Measurement Uncertainty (MU) in Clinical Laboratories.  April 2012.  www.westgard.com/hitchhike-mu.htm

·         Westgard JO. A War of Words in Laboratory Medicine:

o   Part I. Total Error vs Trueness and Measurement Uncertainty.  September 2007.  www.westgard.com/essay116.htm

o   Part II. Concepts and Terminology in a Changing, Uncertain World.  October 2007.  www.westgard.com/w-o-w-partii-concepts-and-terminology.htm

o   Part III. Intended Applications and Customers. October 2007.  www.westgard.com/w-o-w-part-iii-intended-applications-and-customers.htm