Tools, Technologies and Training for Healthcare Laboratories

Quality Goals, Requirements, and Specifications

Everyone agrees that there should be quality control. But what does that mean? When we implement new methods in our laboratories, or develop new methods, or try to establish regulatory guidelines for performance of methods, or even inspect laboratories to assure that good quality management practices have been implemented, how do we specify bias, CV, the number of controls, the levels, etc.? Dr. Westgard introduces a way to sort it all out.

Introduction

In earlier messages on this website, I have pointed out the importance of defining the quality that is needed if we are to manage quality in an objective way. In Jerry Ehrmeyer's guest essay, he talked about quality being relative and suggested that we have to be able to compare what we doing with some "standard." What are the "standards" for the quality of a test result? How should they be stated to be useful in the laboratory? How do we apply them to manage the quality of laboratory testing processes?

Self-assessment exercise

To understand the current difficulties with defining "standards of quality", here's a common management problem. You are looking at a new, more reliable, higher-capacity, larger menu, faster, simpler-to-operate, and less costly analyzer (I read the advertisements too). Let's assume one of the tests to be performed is cholesterol.

  • What method CV and method bias do you want?
  • What control rules and number of control measurements will you use to monitor performance?

These are some of the basic questions that you need to answer to assure the analytical quality of the cholesterol test on this new analyzer.

To help you out, here's some additional information.

  • The US CLIA regulations define 10% as the allowable total error for a cholesterol test [1].
  • The National Cholesterol Education Program (NCEP) specifies an allowable CV of 3% and an allowable bias of 3% [2].
  • For comparison, a European group [3] has defined a precision goal of 2.7% and a bias goal of 4.1% based on the observed individual biological variation which is about 6.5% [4 ].
  • In addition, NCEP provides test interpretation guidelines that recommend values of 200 mg/dl or lower are okay and 240 mg/dl and higher require additional testing to formulate a treatment plan, defining a medically important change or clinical decision interval of 40 mg/dL, which is 20% at medical decision level of 200 mg/dL.

What precision and accuracy do you want? How will you QC this test?

  • CV of ??%
  • bias of ??%
  • control rule(s) such as ??
  • N of ??.

Difficulties with "standards of quality"

These are not intended to be trick questions! These are management decisions that each of us have to make when we a) implement new methods in our laboratories, b) develop new methods for the clinical laboratory marketplace, c) establish regulatory guidelines for monitoring the performance of methods via proficiency testing, and d) inspect laboratories to assure that good quality management practices have been implemented. Regardless of your background and experience, my guess is that you will find it difficult to answer these questions.

Lack of consistent terms

Part of the difficulty comes from the different types of quality goals, criteria for acceptable performance, and performance specifications that are being recommended. Some of these are analytical outcome criteria, such as the CLIA allowable total error in proficiency testing; some are clinical outcome criteria, such as the NCEP medically important change for test interpretation. Others are method performance specifications, such as allowable CV and allowable bias defined by NCEP, and still others are quality goals, such as the recommendations for allowable imprecision and inaccuracy by the European working group.

Trying to understand and compare these different terms is like trying to compare apples, oranges, grapefruit, and bananas - they're all fruit, but they're different kinds of fruit (another example of the fruitbowl approach to laboratory quality management). The allowable total error encompasses both the allowable CV or SD for imprecision and allowable bias for inaccuracy. A medically important change encompasses preanalytical factors, such as the within-subject biological variation, as well as analytical factors such as imprecision and inaccuracy. You will find that certain people, organizations, or agencies have a particular taste. Each selects what they like, which leaves the laboratory with a fruitbowl of choices, rather than a coherent system of recommendations that guide the management of testing processes.

An unstated assumption

From my perspective, most of the recommendations are actually lemons because they provide little practical guidance for the laboratory. They don't work because they are recommendations only for the stable performance of a method, i.e., they assume everything works perfectly and there is no need for quality control because no problems are expected. If this assumption of stable performance is not correct, then it follows that these recommendations are not correct for real laboratories where problems really do occur.

Remember that the analytical quality achieved in the daily operation of a testing process will depend on the both the stable performance of the measurement procedure (i.e., its observed imprecision and inaccuracy) and the capability of the control procedure to detect unstable performance (i.e., changes in imprecision and inaccuracy). Specifications for stable imprecision and inaccuracy are incomplete and inadequate if they fail to consider QC. If you really believe in this assumption of stable performance, it should follow that you don't do any QC! If you perform QC, that's evidence you expect some method problems, therefore, the assumption of perfect method stability is wrong.

Misunderstanding performance as quality

Imprecision and inaccuracy are performance characteristics, not quality requirements. Performance certainly contributes to quality, but they are not the same thing. A given level of analytical quality can be achieved by different combinations of imprecision and inaccuracy. Therefore, setting individual goals for imprecision and inaccuracy in the form of allowable CVs and allowable biases doesn't directly reveal the total error of the testing process that will be experienced by the user or consumer. While calculations can be performed to combine the maximum allowable bias with a multiple of the maximum allowable imprecision to describe the expected total error, such as done by NCEP for lipid tests, these estimates again are flawed because they assume stable performance and don't allow for the performance of the QC procedure.

Relevance to customers

One of the things TQM teaches is that quality is related to customer needs. To determine customer needs for laboratory tests requires communication with physicians and nurses, none of whom really think in terms of precision and accuracy. The cartoon here characterizes the reaction of customers to a laboratory scientist's description of quality and performance. They hear our technical words, but those words don't mean anything to them. For customer communication to work, we have to listen to their words, understand their needs, and translate those needs into our technical terms. This is the process of quality function deployment employed by industry and the key ideas are listening to customers and translating customer needs into process specifications.

Our customers are concerned with the total error that might occur in a test result, not components of errors such as imprecision and inaccuracy. Furthermore, their application of test results is related to certain critical changes from reference values, decision limits, or previous test results. Our customers think about medically important changes in test results; they.don't think about test results with reference to imprecision and inaccuracy. The information available from our customers is in the form of allowable total errors and medically important changes, rather than specifications for allowable imprecision and allowable inaccuracy.

Applicability for laboratory use

The practical purposes of these recommendations for quality goals, criteria for acceptable performance, and performance specifications are to help the laboratory establish, manage, and monitor a testing process to provide the appropriate quality for its patient care services. That means these recommendations should be useful for characterizing the clinical needs of the test, setting purchase specifications for the method, evaluating method performance, establishing internal quality control, and monitoring method performance via external quality assessment or proficiency testing. The real problem is that different types or formats of recommendations are needed at different times and for different purposes in the process for analytical quality management. Each type of recommendation has its place in this system, but the system itself is not well understood.

A Systems View of Analytical Quality Management

The accompanying figure shows the relationships between the various types of quality goals, requirements, and specifications. Starting on the left side of the figure, clinical quality requirements in the form of medically important changes or decision intervals (Dint) can be converted to laboratory operating specifications for imprecision (smeas), inaccuracy (biasmeas), and QC (control rules, N) by a clinical quality-planning model [4] that accounts for preanalytical factors, such as within-individual biologic variability. Biologic goals based on within-subject biologic variability set a boundary condition on operating specifications, defining the most demanding condition for stable performance that would be required to monitor changes in individual subjects. The right side of the figure shows that proficiency testing criteria define analytical quality requirements in the form of allowable total errors (TEa), which can be translated to operating specifications (smeas, biasmeas, control rules, N) via an analytical quality-planning model [5]. Because both the clinical Dint and the analytical TEa can be translated into operating specifications, different types of quality requirements and specifications can be compared with the aid of quality-planning models [6,7].

The most important part of this message is that outcome criteria, such as Dint and TEa, provide the primary quality requirements, which can then be translated into operating specifications that define the performance characteristics required by the testing process. The bottom line in that the laboratory must know the imprecision, inaccuracy, and QC that are necessary to manage and assure the quality of the testing process. Thus, all of these quality goals, criteria, and specifications have some use in the context of a system for analytical quality management. However, they are not independent of each other, and, until this system is recognized and applied, the recommendations in the literature will be incoherent, rather than understandable and consistent.

Clinical quality requirements

Practical information can be provided in the form of a medically important change, medically significant change, or clinical decision limit, which are the commonly used terms for this type of quality requirement. One source of information about medically important changes in test values is a paper by Skendzel, Barnett, and Platt [8]. Note that the important information is found in Table 1 which provides a summary of physicians' opinions of a significant change in test results. This paper is sometimes criticized for the rather large values recommended for medically useful CVs, which appear in Table 2 and were derived without accounting for within-subject biological variation. When Fraser's figures for within- subject biological variation [9,10] are used in a clinical quality-planning model that accounts for biological variation [4 ], OPSpecs charts show that maximum allowable CVs are much smaller than the earlier recommendations [6]. Thus, the original recommendations were limited by an over-simplifed quality-planning model that attributed the total variation to analytical variation, rather than deducting the known biological varation and then determining the allowable analytical variation.

One major advantage of this type of quality requirement is that information is directly available from the customers, either through their description of how they use and interpret a laboratory test, through clinical pathways that detail the expected use and interpretation of tests, or through audits of clinical practices. For example, the NCEP recommendations for test interpretation actually describe differences in test values that would cause changes in diagnosis or treatment. Thus, they define clinical requirements in the form of a decision interval. When this information is properly translated to operating specification via a quality-planning model that accounts for pre-analytical factors, it provides a useful and valid approach for defining and managing the quality of the testing process.

See the table of clinical quality requirements included on this website for a summary of the clinical requirements from Skendzel and the NCEP, as well as some estimates of within-subject biological variation from Fraser and others.

Analytical quality requirements

The most useful form for these requirements is a statement of an allowable total error that encompasses both imprecision and inaccuracy. This corresponds to the industrial "tolerance" specification for process production that considers both the centering of the process on a target value and the distribution of individual products around that target. The most common sources of these type of requirements are the proficiency testing or external quality assessment programs that specify acceptability limits in the form of a target value plus/minus certain tolerances. In the US, CLIA defines such limits for approximately 80 different tests, as shown in the table of analytical quality requirements included on this website. In other countries, such as Australia and Canada, the lists and criteria may be even more extensive.

These PT limits define "minimum" levels of quality that must be achieved, therefore, it is always important to plan testing process that will assure these quality requirements are achieved in routine operation. This can be accomplished by using an analytical quality-planning model that translates these requirements into the imprecision and inaccuracy that are allowable and the QC that is necessary. The practical tool for doing this is again the OPSpecs chart, which can be utilized both for selecting appropriate QC procedures for daily operation and also for judging method performance in initial evaluation studies [11].

Operating specifications

Clinical and analytical quality requirements in the forms of decision intervals and allowable total errors can be translated into practical information needed to manage laboratory testing processes. That information consists of the imprecision and inaccuracy that are allowable for the method and the QC that is necessary to detect unstable performance, i.e., to detect analytical problems that occur with the method. These operating specifications are interdependent and many different combinations can assure the desired quality will be achieved. These many possible combinations can be shown graphically by an OPSpecs charts, allowing managers to quickly and easily determine how to properly manage the analytical quality of a testing process.

Answer to self-assessment exercise

For our cholesterol example, where the NCEP clinical quality requirement is 20% and within-subject biological variation is 6.5%, a clinical OPSpecs chart shows that if method bias were zero, then method CVs of 2.7% or less are needed (see the x-intercepts) if common QC procedures with N=2 are to be used and the false rejection probabilities kept below 0.05 (i.e., a false rejection rate lower than 5%). For a method with a 2.5% CV, appropriate control rules would be the 12.5s or 13s/22s/R4s. For a method with a 2.0% CV, additional choices would be 13s and 13.5s control rules. A method that satisfies the NCEP 3.0% precision and 3.0% accuracy specifications will not assure the desired clinical quality, as can be seen from the operating point of x=3% and y=3% that exceeds the allowable limits of imprecision and inaccuracy for the different QC procedures.


Given the CLIA analytical quality requirement of 10% for cholesterol, an analytical OPSpecs chart shows that if method bias were zero, then method CVs of 2.2% or less are needed if common QC procedures with N=2 are to be used and the false rejection rate is to be kept at 5% or less. For a method with a 2.0% CV, appropriate control rules would be 12.5s or 13s/22s/R4s. For a method with a CV of 1.7% or less, additional choices would be the 13s and the 13.5s control rules. Note that this OPSpecs chart also shows the operating point for a method that satisfies the NCEP 3% imprecision and 3% inaccuracy specifications and that the performance of this method would be judged acceptable in a method evaluation study that used a bias + 2s criterion for stable performance (as shown by the line above the operating point).

Similar conclusions would be reached for a method that satisfied the European quality goals for imprecision of 2.7% and inaccuracy of 4.1%.

In summary, method CVs of 2.0 to 2.5% would generally be required if method bias were zero and simple control procedures with N's of 2 were to be applied.

For more detailed assessments, see the QC planning application for cholesterol using analytical or clinical quality requirements.

For more discussion of OPSpecs charts, click here.

References

  1. U.S. Department of Health and Social Services. Medicare, Medicaid, and CLIA Programs: Regulations implementing the Clinical Laboratory Improvement Amendments of 1988 (CLIA). Final Rule. Fed Regist 1992(Feb 28);57:7002-7186.
  2. National Cholesterol Education Program Laboratory Standardization Panel. Current status of blood cholesterol measurement in clinical laboratories in the United States. Clin Chem 1988;34:193-201.
  3. Fraser CG, Hyltoft Petersen P, Ricos C, Haeckel R. Proposed quality specifications for the imprecision and inaccuracy of analytical systems for clinical chemistry. Eur J Clin Chem Biochem 1992;30:311-7.
  4. Westgard JO, Hytoft Petersen P, Wiebe DA. Laboratory process specifications for assuring quality in the U.S. National Cholesterol Education Program (NCEP). Clin Chem 1991:37:656-661.
  5. Westgard JO, Wiebe DA. Cholesterol operational process specifications for assuring the quality required by CLIA proficiency testing. Clin Chem 1991;37:1938-44.
  6. Westgard JO, Seehafer JJ, Barry PL. Allowable imprecision for laboratory tests based on clinical and analytical test outcome criteria. Clin Chem 1994;40;1909-14.
  7. Westgard JO, Seehafer JJ, Barry PL. European specifications for imprecision and inaccuracy compared with operating specifications that assure the quality required by U.S. CLIA proficiency testing criteria. Clin Chem 1994;40:1228-32.
  8. Skendzel LP, Barnett RN, Platt R. Medically useful criteria for analytic performance of laboratory tests. Am J Clin Pathol 1985;83:200-5.
  9. Fraser CG. Biological variation in clinical chemistry. An update: collated data, 1988-1991. Arch Pathol Lab Med 1992;116:916-23.
  10. Fraser CG. The application of theoretical goals based on biological variation data in clinical chemistry. Arch Pathol Lab Med 1988;112:404-15.
  11. Westgard JO. A method evaluation decision chart (MEDx Chart) for judging method performance. Clin Lab Science. 1995;8:277-83.