Quality Requirements and Standards
Back to the Future: 2014 Quality Goals sound a lot like 1976
The current (2014) discussion on Quality Goals seems a lot like the debate as it stood nearly 40 years ago. To see why current Quality Goal thinking may only be a "revolution" in the most literal sense, read this flashback to the 1970's
Analytical Goals in Clinical Chemistry – 1976 Aspen Conference
James O. Westgard, PhD
December 2014
The current discussion on quality goals takes me back to the 1st conference that was held on this topic in 1976. Known as the Aspen Conference, it was organized by the College of American Pathologists and included 30 participants, mainly from the US, but also individuals from England and Denmark. Later on, a similar conference was held in England, sponsored by international pathology organizations.
Aspen recommendations
The Aspen conference report provides some interesting recommendations that are echoed in more discussions in the Stockholm and Milan meetings.
“During the past 30 years, three major approaches have been used to define analytical goals in clinical chemistry: 1) Intuitive relationships to the normal range or biologic variability; 2) interlaboratory testing criteria; 3) medical significance criteria…
“The outstanding achievement of this meeting is the development of an integrated theoretical basis for defining analytical goals for individual chemical analysis…
“The 1976 Aspen Conference on Analytical Goals in Clinical Chemistry utilizes the familiar parameters of analytic variance and biological variance, but assigns to them new significance as quantitative terms in formulae which are relevant in three possible medical situations: 1) group screening; 2) individual single point testing, and 3) individual multipoint testing (trend analysis).
Biologic goals adopted!
“For group screening in which an individual is to be selected from a population, a goal for an analytic coefficient of variation (CVa) is defined as
CVa = 0.5 [(CVintra-individual)2 + (CVinter-individual)2]1/2
“For individual single and multipoint testing in which an individual is evaluated on the basis of discrimination values, a goal for an analytic coefficient of variation is defined as
CVa = 0.5 CVintra-individual
Biologic variation was the “new” goal setting model that was adopted based on work by Cotlove, Harris, and Williams at NIH and Statland and Winkle in Denmark. Gene Harris provided much of the statistical framework to support the derivation and application of biologic goals. It should be noted that Bernie Statland had completed a post-doc at NIH before moving to Denmark to work with Paul Astrup and Per Winkel, where they carried out many studies of the influence of biologic factors on the variability of test values. The point is that the first use of biologic variability for setting analytic goals occurred in the US, whereas later studies on biologic variability and applications of biologic goals were more widely carried out in Europe. Today we regard biologic goals as “European” goals, but they have a US origin.
Bias not considered!
“There is a major need for further study and understanding of the statistical relationships that should be used to derive a clinical diagnosis from an analytic value. The analytical goals for these analytes have been based upon a statistical model which considers only random variability. The model allows a maximum of about 12% increase in observed variability which can be attributed to random analytic variability. Other statistical models which offer alternative goals, including consideration of accuracy, and which are based on other assumptions, should be investigated with the view that more perceptive and useful techniques may be developed.”
No goals were set for bias! In fact, it was argued that bias could be ignored because laboratories [at that time] developed their own reference ranges, hence bias was supposed to be accounted for by consistent systematic error in test values and reference ranges. However, the later European conference did acknowledge that accuracy (bias) needed to be considered and that quality goals for bias were needed.
As a young clinical chemist, I argued for the need to consider bias based on the fact that there were biases between different methods, that laboratories often had more than one method in use, and that reliable reference ranges studies were difficult to perform in all laboratories. However, the idea that quality was the result of a combination of bias and precision ran into strong opposition from the established pathology community. Nonetheless, the concept of total error was introduced into the discussion of quality goals and became more widely debated in the clinical chemistry and pathology community.
Need for Total Error goals today!
Like it or not, proficiency testing (PT) and external quality assessment (EQA) surveys are required by virtually all regulatory and accreditation programs. Like it or not, criteria for acceptable performance must be defined to “score” participants in such programs. Like it or not, those criteria implicitly are total error criteria because virtually all PT/EQA programs are based on a single measurement of each survey sample. Like it or not, that single measurement may be in error due to both random error (imprecision) and systematic error (inaccuracy, bias), thus the laboratory results reflect total errors in measurement, not just precision.
It is important to understand that the argument being made by Oosterhuis (in the recent Milan conference) is that the goal-setting model for calculating an allowable Total Error from biologic variation data is wrong, particularly how the maximum allowable bias is combined with the maximum allowable CV to calculate an allowable Biologic Total Error. That doesn’t mean it is wrong to employ Total Error criteria for evaluating the quality of a method, characterizing quality on the sigma-scale, selecting SQC procedures, and evaluating PT/EQA results. Rather, it means that you have to be careful about how you define the allowable Total Error that is being used as the goal! Here’s a good example of the old adage ‘Don’t throw the baby out with the bath water.” The bath water may be dirty, but the baby may be clean!