Quality Requirements and Standards

Another slew of MU has hit the journals. The MU-TE debate continues in publications and online postings. But there are some reasons for hope...

Making Quality Great Again!  I'm with Quality!

Some dismay, but less disagreement, on MU vs. TE

James O. Westgard, PhD, and Sten Westgard, MS
June 2016

To the laboratory professional, the ongoing debate about “MU vs. TEA” must sometimes resemble an endless squabble, with arguments recycled from past decades and rhetoric replacing real data on utility and application. So long as TEA exists, there are some in the MU-vement that salivate over its possible demise. Let’s be clear: this isn’t a fight chosen by TEA advocates – we believe MU and TEA should coexist and complement each other. But for the pure MU-trologists, there seems no possible compromise. TEA must be terminated and the past 40 years of laboratories applying and implementing TEA is an aberration that must be undone. For labs seeking practical guidance on QC and method performance, unfortunately, the immediate outcome of this fight is counter-productive, a scientific circular firing squads of sorts. The real enemy is the lack of quality provided by manufacturers and vendors, not the wording of the definition of quality.

The latest round in the debate, however, seems to be more slightly more hopeful. A recent opinion paper [1] raises a concern about our assessment of the ISO 15189 guidance for measurement uncertainty (MU) and the consequences for its determination and validation.

“It is with some dismay we have observed a growing debate with regard to the use of ‘Westgard style’ total error and ‘GUM style’ uncertainty of measurement [2,3]. The recent release of an article on the well-regarded Westgard Internet site adds to this debate but appears to distort some of the issues involved [4]. As such, we feel obliged to present an alternate view and to outline how the issues described by Westgard have been successfully addressed in Australia with the ISO 15189 accreditation [5].”

[Please note: “Westgard Style” up until this point, has always been defined as elbow patches on your jacket and a well-trimmed mustache.]

Less disagreement

We applaud the Australian efforts to implement ISO 15189 accreditation and their practical approaches to resolving issues with MU. Their leadership and forward thinking provides guidance for everyone and we agree with most of their points:

• TAE and MU are not incompatible, which is why we recommend that both be included in a comprehensive Quality Management System [3].
• A major problem with acceptance of the GUM methodology is the complex statistical calculations. The GUM recommendations for MU have been around for some 20 years and MU has been included in ISO 15189 for some 10 years (though not required until the 2012 edition). Yet few laboratories today make any practical use of MU – see the global MU survey results - except to satisfy an accreditation requirement.
• We are grateful for the clarification that GUM section F.2.4.5 “specifically describes the procedure where a significant systematic effect may be taken into account by enlarging the uncertainty assigned to the result... This situation is directly comparable to the calculation of Total Error as described by Westgard and others, where the uncertainty includes both an imprecision and bias component...” This little known clause is seldom mentioned nor referenced in arguments about TAE vs MU.
• We also prefer the separate experimental determination of bias and imprecision. The sources of experimental data vary over the life cycle of a measurement procedure, beginning with the initial validation data, then progressing to ongoing SQC data for estimates of imprecision, PT and EQA surveys for estimates of bias, as well as periodic use of certified reference materials and trueness controls.
• We also agree that goals for acceptable performance can be established based on biologic variability, following the recommendations of Fraser [6,7] for calculating allowable bias and allowable CV and utilizing the database established by Carmen Ricos and her Spanish colleagues [8]. This is an example of the Australians’ practical adaptation of existing error goals to set targets for MU, but this position is not as well accepted in other countries.

More Dismay

However, our understanding of MU and TAE and interpretation of ISO 15189 differ from the Australians on a few issues.

• Our statement that ISO recommends a practical approach to determine MU is based on the guidance in section 5.5.1.4 Note 2, which states that “measurement uncertainties may be calculated using quantity values obtained by the measurement of quality control materials under intermediate precision conditions that include as many routine changes as reasonable possible in the standard operation of a measurement procedure, e.g. changes of reagent and calibrator batches, different operators, schedule instrument maintenance.” It is true that Note 2 says “may” and not “shall”, which means laboratories do not have to do it this way, but many laboratories find this guidance much more practical than following the complex GUM methodology. Note 2 provides a major change from the earlier 2007 draft that recommended MU be determined when “relevant and possible.” Most laboratories considered that MU was not relevant and the GUM methodology was not possible to implement, therefore few labs implemented MU. That’s probably why the 2012 revision provides more practical guidance for using existing SQC data to calculate an SD that can be used to estimate MU.
• The authors note that ISO 15189 does not include GUM as a normative reference, therefore professional judgment allows the use of Total Analytic Error as an alternative to MU, or any other methodology for that matter, as long as it is defined and documented. That position represents the forward thinking of the Australians, a sharp and intriguing viewpoint, but this view is not typical of the metrologists who advocate for MU implementation in other countries. GUM does appear as a reference (number 56) in the ISO 15189 document and if laboratories are looking for guidance for fulfilling the requirement for MU, the existing reference list points to GUM. About 30 CLSI documents are cited, but there is no reference to EP21 [9] for estimation of TAE.
• We disagree with the Australian statement that “bias becomes largely irrelevant provided it remains internally consistent and consistent with the comparator,” even though “in this situation both total error and MU essentially become an assessment of imprecision.” While true in principle, in practice we see widespread and significant between-method biases in labs around the globe, wit differences large enough to affect the accuracy and interpretation of test results, particularly for comparison with nationally or globally established clinical treatment guidelines. It is therefore a worrying practice to assume a test result will only be compared with another test result from the same instrument or method (comparator) or to reference intervals that have been carefully established in individual laboratories. In most of the labs we visit and work with, this is not true.
• Given that GUM provides some credence for the TAE approach, the statement that TAE is a “statistically contrived concept” lacks scientific rationale. On the one hand, the Australians note that TAE is referenced by GUM and very practical to implement, but on the other hand, they dismiss the scientific foundation. But in contrast to the complex GUM methodology, TAE offers a simple, practical measure of the ISO concept of accuracy, which is defined as “closeness of agreement between a test result and the accepted reference value; Note: The term ‘accuracy’ when applied to a set of test results, involves a combination of random components (imprecision) and a common systematic error or bias.” TAE provides a 95% interval of the expected analytic error, similar to expanded measurement uncertainty with a coverage factor of 2. The difference is that this error interval is shifted in the direction of the observed bias, rather than being centered on the test result.

[Ouch: that’s quite the sting. Using both “statistically” and “contrived” in the same phrase. We all hate statistics, and to add insult to injury, we’re now contriving the term. Total error is a concept that was created to represent a specific impact of analytical errors. But basically every other statistic is also contrived, created, invented and derived by some scientist for (it would be hoped) some practical use. MU, at this point is very contrived – and seems to get more contrived with every new twist, tweak, adjustment and publication.]

The Same Old Politics

We too are getting tired of these ongoing arguments about MU vs TAE. We too think that both concepts have their uses and applications. TAE has been applied in medical laboratories for over 40 years and is well accepted globally. Our belief is that TAE is most useful for managing the analytical quality in the medical laboratory, assessing the acceptability of performance, determining quality on the sigma scale, designing SQC procedures to verify the attainment of the intended quality of test results, developing risk-based QC Plans, and monitoring the sigma quality of field methods through PT and EQA programs [10]. MU is most useful for manufacturers in planning, developing, and producing better measurement procedures by identifying and eliminating sources of errors. Both concepts may be expanded to include other pre-analytic variables, such as sampling variation, sampling bias, within-subject biologic variation, effects of multiple samples vs replicate measurements, etc., thus providing better information for the use and interpretation of test results. Some professionals now view the uncertainty concept as mainly useful for assessing diagnostic uncertainty and are attempting to redefine MU to focus on optimizing laboratory tests for better interpretation of test results [11-13]. This may further complicate the application of the uncertainty concept, but ultimately may be a better intended use.

As with politics, many in the laboratory community see the debate and the arguments and become discouraged, disinterested and drop out of the discussion. This the exact opposite of what we need: we need more labs with more engagement, more data, more studies, and more commitment to quality. When we stop caring about the debate, we surrender to the indifference and imprecision and inaccuracy. We begin to passively accept whatever the manufacturers give us, rather than challenge them to provide better performance.

And those who suffer most are those who have no voice in this debate at all: the patients.

References

1. Farrance I, Badrick T, Sikaris KA. Uncertainty in measurement and total error – are they so incompatible? Clin Chem Lab Med online, May 2016, DOI 10.1515/cclm-2016-0314.

2. Panteghini M, Sandberg S. Total error vs measurement uncertainty: the match continues. Clin Chem Lab Med 2016;54:195-6.

3. Westgard JO. Useful measures and models for analytical quality management in medical laboratories. Clin Chem Lab Med 2016;54:223-33.

4. Westgard S. Quality requirements and standards – quality goals at the crossroads: growing, going, or gone? Available at http://www.westgard.com/gone-goals-gone.htm

5. International Organizations for Standardizations. ISO 15189:2012, Medical laboratories – Requirements for quality and competence. Available at http://www.iso.org/iso/catalogue_details?csnumber=56115.

6. Fraser CG, Petersen P. Quality goals in external quality assessment are best based on biology. Scand J Clin Lab Invest 1993;53(Suppl 212):8-9.

7. Fraser CG. Biological Variation: From principles to practice. Washington DC:AACC Press, 2001.

8. Ricos C, Alverez V, Cava F et al. Current databases on biologic variation: pros, cons, and progress. Scand J Clin Lab Invest 1999;59:491-500.

9. CLSI EP21. Estimation of Total Analytic Error for Clinical Laboratory Methods. Wayne PA:Clinical and Laboratory Standards Institute, 2003.

10. Westgard JO, Westgard SA. A graphical tool for assessing quality on the sigma-scale from proficiency testing and external quality assessment surveys. Clin Chem Lab Med 2015;53:1531-6.

11. Oosterhuis WP, Theodorsson E. Total error vs measurement uncertainty: revolution or evolution. Clin Chem Lab Med 2016;54:235-239.

12. Jones GRD. Measurement uncertainty for clinical laboratories – a revision of the concept. Clin Chem Lab Med DOI 10.1515/cclm-2016-0311.

13. Tate JR, Plebani M. Measurement uncertainty – a revised understanding of its calculation and use. Clin Chem Lab Med DOI 10.1515/cclm-2016-0327.