Quality Requirements and Standards

After nearly 8 years of building the new EFLM biological variation database, the recommendations for anaytical quality specifications have been adjusted downward. Desirable specifications are too demanding. What should labs do instead? The minimum.

Perspectives on Analytical Quality Management

Part 1. TAE Model for Quality Assessment/Control vs TEa Model for Goal-Setting

James O. Westgard, PhD
July 2022

Ring the bells that still can ring
Forget your perfect offering
There is a crack in everything
That’s how the light gets in.”
—Anthem (1992) by Leonard Cohen

We published a paper in 1974 about “Criteria for judging precision and accuracy in method development and evaluation” [1] that introduced the concept of Total Analytical Error (TAE model) to account for both the precision and bias of a method when judging the acceptability of method performance. The accepted practice at that time was to consider precision and bias as separate characteristics. Typically, the comparison of results between methods was assessed using the correlation coefficient, which (as you should know) is not affected by systematic errors between methods. We had studied the behavior of statistical tests in method comparison studies earlier [2] and recognized that regression analysis was necessary to properly identify and quantify the proportional and constant components of systematic errors that together made up the bias of a method. Bias was typically determined by t-test analysis as an estimate of the combined systematic error at the mean of the comparison data. Estimates of bias at specific decision levels could be made using the observed regression coefficients.

After publishing these two papers in the journal of Clinical Chemistry, we presented recommendations for evaluating methods at the 1976 Aspen Conference on Analytical Goals in Clinical Chemistry and emphasized the need for a Total Analytical Error criterion as an analytical goal [3]. The Aspen Conference was sponsored by the College of American Pathologists and involved some 30 leading clinical pathologists and clinical chemists [4]. The notable outcome was a consensus to use goals for precision based on intra-individual and inter-individual biologic variation. Today you would never know that the origin of the approach for using biologic goals was US laboratory scientists as the European Federation of Laboratory Medicine has assumed ownership of the biologic database and methodology for setting Analytical Performance Specifications. However, the pioneer work was done at NIH by Ernest Cotlove and George Williams [5], after which Bernard Statland and Eugene Harris were the major movers who developed this approach. Statland served as a Post-Doc at NIH and Harris was the statistician who worked with Cotlove and Williams. In addition, Statland worked with Per Winkel, a clinical chemist and statistician from Copenhagen, and their work stimulated an interest in biologic variability in Europe.

I met Bernie in Copenhagen at that time while on a trip to attend a conference in France. Bernie was then working in Poul Astrup’s laboratory and writing a prodigious number of papers about the factors influencing biologic variability. By coincidence, it was on this trip that I met Professor Carl-Henric de Verdier from Uppsala University. Svante Wold, a collaborator on the TE paper when he worked in Madison with Dr. George Box in the Statistics Department, was also from Uppsala University, so the connections between Madison and Uppsala were strengthened and led to my taking a sabbatical leave to study in Uppsala in 1976-7.

Aspen Conference CAP Goals for Precision

The outcome from the CAP conference was disappointing in that the analytic goals focused only on precision. The conference report stated the following:

“The conference participants have taken the position that there is sufficient documentation of biological variability for the analyses under consideration that analytic variance can be defined as a fixed fraction of biologic variation. Therefore, the goal was established that analytic variance should not exceed one-quarter of biologic variance. In this way, analytic variance will contribute no more than 11.8% to total observed variability…”

What about accuracy or bias? There were no recommendations for such goals. The assumption of many at the conference was that the laboratory could correct for bias by defining normal ranges that reflected any method bias. In fact, there were some objections to even considering the accuracy of methods.

“…I must say that I take strong objection to the idea that you place so much emphasis on accuracy.”

That comment from one of the deans of pathology ended any discussion on goals for accuracy or bias. I had argued that current laboratories often had two or more methods for measuring certain constituents and therefore shouldn’t just ignore systematic differences between methods, but the consensus was that precision was the important performance characteristic and bias was not really an issue. A year later, when CAP worked together with clinical pathologists in the UK to expand the discussion of analytical goals, their recommendations acknowledged that goals for accuracy may also be important.

Thus, the introduction of a criterion to judge the acceptability of the observable Total Analytical Error was broadly opposed because it took bias into account and required laboratories to deal with systematic differences between methods. However, over time, TAE became widely accepted by clinical laboratory scientists because of the growth of PT and EQA programs. Those PT/EQA programs generally allowed for only one measurement on a survey sample, thus any survey result was affected by both the random and systematic errors of the method in use by the laboratory. The growth of such PT/EQA programs was global and the grading of results became more and more quantitative, leading to tables/lists of the acceptable performance in terms of allowable Total Errors. The use of biologic variability to define goals for allowable TE in EQA programs came later.

Total Confusion about Total Error Models

In a recent exchange of ideas [6-8], we commented on a proposal to change IQC practice by employing the Analytical Performance Specification for Measurement Uncertainty (95% range) directly as control limits on control charts. We have had a number of previous discussions of the relative merits of different quality management approaches, particularly the Total Error/Six Sigma framework vs the Measurement Uncertainty/Traceability framework [9-11].

The latest discussion focuses on a recommendation by metrologists to re-design laboratory SQC to employ two different control procedures [12]. Component I is like traditional SQC in that its objective is to evaluate the daily performance of the testing process; Component II uses an additional control solely for the purpose of estimating measurement uncertainty (MU). We confined our objections to the recommendation to use the Analytical Performance Specification for MU (95% limits) as the control limits on control charts. This violates a fundamental principle of Statistical QC that control limits should be determined based on the observed means and SDs in the laboratory.

TAE Model for QC Planning.

To establish proper QC limits, we recommended following the guidance from CLSI C24-Ed4 and its “roadmap” for planning SQC strategies and determining the appropriate control rules, numbers of control measurements, and frequency of QC events. The process for doing this has been described in CLSI C24-Ed4 [13] by the following steps:

1. Define the quality specifications for the test.
2. Select appropriate control materials
3. Determine the stable (in control) performance characteristics of the measurement procedure.
4. Identify candidate quality control strategies.
5. Predict QC performance.
6. Specify desirable goals for the QC performance characteristics.
7. Select a quality control strategy (control rules, number of control measurements) whose predicted performance meets or exceeds the quality control performance goals.

Unfortunately, metrologists reject the C24-Ed4 approach because it employs a quality requirement in the form of an allowable Total Error. The application of a TEa goal is criticized because it takes bias into account. According to the theory of metrology, bias should be eliminated, corrected, or ignored because its existence undermines the fundamental assumption and methodology to characterize measurement uncertainty, which metrologists favor as the proper way to measure analytical performance.

TEa Model for Goal-Setting.

An argument commonly made by metrologists is that the model for setting of goals for allowable Total Error (TEa model) based on biologic variability is wrong. As stated by Panteghini, “I would like just to remember that the common model employed to derive a limit for the allowable TE uses a mathematically incorrect method relying on the sum of mutual exclusive terms.” He references this statement to Oosterhuis [14], who in turn references the methodology for calculating allowable bias from biologic variability to Gowens et al [15] and for calculating allowable Total Error from the biologic goals for bias and imprecision to Fraser and Hyltoft Petersen [16], who introduced the equation that combined the maximum allowable bias and the maximum allowable SD or CV to provide a single performance measure in the form of TEa.

The point is - this TEa model for calculating analytical goals based on biologic variation is not our model. It is a European goal-setting model. Our TAE model provides quantitative criteria for assessing the precision, bias, and total error during method development and evaluation, then extends to planning SQC procedures and strategies for monitoring quality during routine operation. Our TAE model is NOT a goal-setting model. While we have tabulated sources of such quality goals to make that information more easily available to our laboratory users, we did not formulate those equations for calculating allowable errors from data on biologic variation.

The European goal-setting model was utilized by Ricos and colleagues [17] in their database for biologic goals that was published in the proceedings of the 1999 Stockholm Conference on “Strategies to set global analytical quality specifications in laboratory medicine” [18]. Publication of the proceedings of that conference popularized the use of biologic variability for setting analytical performance goals. Following the Milan Conference in 2015 [19] to update the Stockholm recommendations, the European Federation for Laboratory Medicine has taken over the methodology and data base for determining biologic goals, thus this TEa goal setting model now resides with EFLM. It is their purpose to establish Analytical Performance Specifications (APS) for laboratory methods based on biologic variability. So, metrology advocates should remember that the goal-setting problem resides in-house in EFLM and it is theirs to eliminate, correct, or ignore, but it is a gross mis-representation to blame us for this problem.

References

1. Westgard JO, Carey RN, Wold S. Criteria for judging precision and accuracy in method development and evaluation. Clin Chem 1974;20:825 33.
2. Westgard JO, Hunt MR. Use and interpretation of common statistical tests in method comparison studies. Clin Chem 1973;19:49 57.
3. Elevitch FR, Batsakis JG, Barnett RN, Gilbert RK, Grannis GF, Peters T. Proceedings of the 1976 Aspen Conference on Analytical Goals in Clinical Chemistry. College of American Pathologists, Chicago IL, 1977.
4. Westgard JO. The development of performance standards and criteria for testing the precision and accuracy of laboratory methods. Proceedings of the 1976 Aspen Conference on Analytical Goals in Clinical Chemistry. College of American Pathologists, Chicago IL, 1977.
5. Cotlove E, Harris EK, Williams GA. Biological and analytical components of variation in long-term studies of serum constituents in normal subjects. 3. Physiological and medical implications. Clin Chem 1970; 16:1028-1032.
6. Westgard JO, Bayat H, Westgard SA. How to evaluate fixed clinical QC limits vs. risk-based SQC strategies. LTE: Clin Chem Lab Med 2022; https://doi.org/10.1515/cclm-2022-0539.
7. Panteghini M. Reply to Westgard et al.: ‘Keep your eyes wide… as the present now will later be past.’ LTE: Clin Chem Lab Med 2022; https://doi.org/10.1515/cclm-2022-0557.
8. Plebani M, Gillery P, Graves RF, Lackner KJ, Lippi G, Nelichar B, Payne DA, Schlattmann P. Rethinking internal quality control: the time is now. Clin Chem Lab Med 2022; https://doi.org/10.1515/CCLM-2022-0587.
9. Westgard JO. Managing quality vs measuring uncertainty in the medical laboratory. Clin Chem Lab Med 2010;48: 31-40.
10. Westgard JO. Useful measures and models for analytical quality management in medical laboratories. Clin Chem Lab Med 2016;54:223-33.
11. Westgard JO. Error methods are more practical, but uncertainty methods may still be preferred. Clin Chem 2018;64:636-8.
12. Braga F, Pasqualetti S, Aloisio E, Panteghini M. The internal quality control in the traceability era. Clin Chem Lab Med 2021;59:291-300.
13. CLSI C24-Ed4. Statistical Quality Control for Quantitative Measurement Procedures: Principles and Definitions. 4th ed. Clinical and Laboratory Standards Institute, 9950 West Valley Road, Suite 2500, Wayne PA, 2016.
14. Oosterhuis WP. Gross overestimation of total allowable error based on biological variation. Clin Chem 2011;57:1334-6.
15. Gowans EMS, Hyltoft Petersen P, Blaabjerg O, Horder M. Analytical goals for the acceptance of common reference intervals for laboratories throughout a geographic area. Scand J Clin Lab Invest 1988;48:757-64.
16. Fraser CG, Hyltoft Petersen P. Quality goals in external quality assessment are best based on biology. Scand J Clin Lab Invest 1993;53(Suppl212):8-9.
17. Ricos C, Alvarez V, Cava F, Garcia-Lario JV, Hernandez A, Jimenez CV, Minchinela J, Perich C, Simon M. Current databases on biological variation: pros, cons and progress. Scand J Clin Lab Invest 1999;59:491-500.
18. Hyltoft Petersen P, Fraser CG, Kallner A, Kenny D. Strategies to Set Global Aanalytical Quality Specifications in Laboratory Medicine. Scand J Clin Lab Invest 1999;57(No 7), 475-583.
19. Panteghini M, Sandberg S. Defining analytical performance specifications 15 years after the Stockholm Conference. Clin Chem Lab Med 2015;53:829-832.