June 2000
An updated version of this essay appears in the Nothing but the Truth about Quality manualJames O. Westgard, PhD, FACB
A conference on "New Approaches to Quality Control" was held in Chicago May 11-12, 2000. Sponsored by the American Association for Clinical Chemistry (AACC) and the British Association for Clinical Biochemistry (ACB), this conference is also scheduled for September 28-29 in Cambridge, UK. The day and a half of presentations were strong on theory and ideas, but weak on practical guidelines and tools for implementation. I won't make any attempt to cover all the presentations in this review. My intent is to summarize some of the important recommendations, offer some insights into the practical application of these new approaches, and describe how to support the planning and implementation of improved quality systems in healthcare laboratories.
I'll begin with one of the last speakers, Judith Yost, who is Director of the Division of Laboratories and Acute Care Services at the Health Care Financing Agency (HCFA). She spoke about the final, final, final, final, final CLIA regulations that will hopefully be published before the end of the year 2000. While unable to provide details of the specific language in the final regulations until they're published in the Federal Register, she did state there was no provision for clearance of manufacturer's QC instructions by the Food and Drug Administration (FDA). QC clearance was a major component of the original regulations and has been postponed again and again and now postponed forever. Finally it's clear that government clearance of manufacturer's QC instructions won't occur!
Why is this so important? Up till now, the QC guidelines for moderately complex methods stated that laboratories could follow the manufacturer's QC instructions if those instructions were cleared by the FDA. In the absence of such clearance, those methods would be treated as highly complex methods that are subject to more stringent QC guidelines. For highly complex methods, "the laboratory must evaluate instrument and reagent stability and operator variance in determining the number, type, frequency of testing calibration and control materials and establish criteria for acceptability used to monitor test performance during a run of patient specimen(s)" [see reference 1, paragraph 493.1218(b)].
Will laboratories now be held responsible for determining appropriate quality control? If so, how should laboratories take into account the instrument and reagent stability and operator variance in establishing their QC procedures? What should laboratories do to provide the appropriate number, type, and frequency of testing of calibration and control materials? How should laboratories establish appropriate criteria for monitoring routine runs?
In the discussion that follows, I'll show you how the new QC approaches can be used to improve quality control and also address the CLIA issues for establishing appropriate QC procedures.
The importance of relating QC to defined quality goals was discussed by Dr. Carl Garber, who is Corporate Director of Statistical Applications for Quest Diagnostics. Dr. Garber provided an overview of the process for establishing and managing a laboratory testing process and showed how quality goals should first be used in validating the imprecision and inaccuracy of analytical methods and then for selecting appropriate QC procedures. This approach certainly addresses the CLIA requirement for evaluating instrument and reagent stability and operator variance in establishing QC procedures to monitor routine performance. A more detailed planning procedure is provided by NCCLS's recent update of its guidelines for statistical quality control [2].
Dr. Garber also described how information about the total testing process was integrated into a "multi-component statistical process control" computer program called SmarTechTM, which is being deployed throughout Quest laboratories. This program makes use of control results, patient results, sample conditions, calibration information, patient demographics, and other rule-based information. While this program is proprietary and not available to other laboratories, it illustrates many of the capabilities that are needed in new QC technology.
The need for a new theoretical model to deal with new technology was addressed by Dr. Curtis Parvin, who is the Director of Informatics and Statistics in the Division of Laboratory Medicine at Washington University in St. Louis. The difficulty of defining an analytical run is central to his approach, which focuses on "event-driven QC" and "non-event QC". In Dr. Parvin's terminology, event-driven QC refers to performing QC when something is known to have happened or to have been changed, i.e., an event is known to occur and QC must be performed to determine whether that event has affected the quality of the testing process. Non-event QC is the monitoring of performance during routine operation to detect unknown events.
From my perspective, event-driven QC is similar to our earlier use of "startup QC" and non-event QC is similar to our use of "monitor QC" in an approach we called "multistage QC" when it was published in 1987 [3]. Multistage QC evolved from a strategy of "nested stages of control" that was presented in a paper in 1984 [4]. What is new from Dr. Parvin is the statistical model for predicting QC performance and the new terminology. Our earlier model, which was called the traditional model by Dr. Parvin, bases the selection of QC procedures on maximizing the probability of error detection and minimizing the probability for false rejection. The traditional model still seems appropriate for event-driven QC where the objective is to select a QC procedure with high error detection. For non-event QC, Dr. Parvin's model utilizes a measure of defect rate, which leads him to conclude that the regular spacing or placement of controls is more important than the control rules that are used.
In response to a question from the audience, Dr. Parvin stated that the intent in monitoring with stable control materials is only to detect large errors and that any control rule will do. To detect smaller errors, he recommended using patient data QC procedures. This is an interesting recommendation because it means that the new theoretical model is not actually needed if any QC rule will do. Certainly the monitoring QC procedure should have a low false rejection rate, otherwise there would be needless rejections, trouble-shooting, and repeat runs. Certainly more error detection is better than less error detection, otherwise why use any QC procedure at all! Furthermore, if you can't implement a patient data QC procedure to detect small errors, then you have to accomplish this with your monitoring procedure that utilizes stable control materials.
This was the most complicated presentation at the meeting and the program chairman remarked that he'd heard it three times and was only now beginning to understand it. To boil it down to what's the essential for general QC applications, the recommendation is to implement three different QC procedures, one having high error detection procedure for event-driven QC, a second using regularly spaced controls to detecting large errors while monitoring method performance, and a third using patient data QC for detecting smaller changes during the monitoring phase.
The theme of using patient-data QC to complement traditional reference-sample QC was developed by Dr. George Cembrowski, who is Director of Medical Biochemistry at the University of Alberta Hospital. He described the use of "Average of Normals" (AoN) or "Average of Patients" (AoP) procedures in which the patient results are first trimmed to eliminate obvious abnormal values, then averaged to provide a statistic for monitoring systematic changes. Dr. Cembrowski developed the methodology for selecting an appropriate number of patient specimens on the basis of the ratio of population variation to analytical variation in a landmark paper published in 1984 [5]. What is new is a recommendation to use exponentially smoothed means rather than traditional averages as the statistical control parameter. Dr. Cembrowski was the first to describe the use of exponential smoothing for reference-sample QC data in a paper on trend detection in 1975 [6]. While the actual techniques are old, there seems to be a new opportunity for applying AoN procedures today because of improvements in computer systems. The difficulty of defining run length could actually be addressed by measuring the stable period of operation using an AoN procedure [7].
The impact of analytical errors on the medical classification of patients was discussed by
Dr. George Klee of the Mayo Clinic. He showed that very small systematic errors could cause large changes in the number of patients who were classified into different diagnostic groups, particularly when following clinical guidelines that have a single cutoff point. Dr. Klee emphasized the need for very precise analytical methods to maximize the error detection capability of QC procedures, but pointed out that the QC procedures themselves should be aimed at detecting systematic errors.Traditionally, QC procedures have been optimized for detection of systematic errors for many years, as illustrated in detail in 1990 by the design of QC for tests performed on an automated chemistry analyzers [8]. Current design tools, such as the QC Validator computer program, allow the user to choose whether to optimize for systematic (SE) or random errors (RE), but the default setting is SE and that is the recommended method of optimization [9,10].
What is new in Dr. Klee's approach is his use of "tolerance limits for analytic bias" as the quality goal, rather than a total allowable error or clinical decision interval. His design objective is to maintain the same proportions of patients who are classified into different diagnostic groups. When this is achieved, physicians shouldn't notice any significant changes in the number of patients being classified in a certain group. However, individual patients may be incorrectly classified because of analytic or biological variability. It seems like the quality of laboratory testing is being defined as getting the correct results for the group as a whole, rather than the correct result for each individual patient!
In any case, Dr. Klee's analytical tolerance limits lead to very tight specifications for method performance. For cholesterol, for example, Dr. Klee stated that methods should have a CV of 1% or better, which can be compared to NCEP's recommendation of 3% or better [11]. Our own assessment has been that a CV of 2.0% or better is necessary to satisfy CLIA analytical performance requirements [12], which turns out to be more demanding than the NCEP physician guidelines for interpretation of a cholesterol test [13].
An approach for identifying different quality monitors for different sources of errors was presented by Dr. Paula Santrach, who is Co-Director of Hospital Clinical Laboratories at Mayo Clinic and vice-chair of the NCCLS committee that developed the quality-system approach for unit-use devices [13]. This approach utilizes a "source of errors matrix" to identify potential errors and define an applicable quality monitor or "method of control." The NCCLS EP18P document provides an example sources of error matrix with over 100 possible error sources as a starting point. The resulting quality system makes use of the series of methods of control that are appropriate for the particular device. The methods of control often include operator training and proficiency, which are seldom quantitative and often difficult to implement.
Dr. Santrach suggested that this quality systems approach was applicable to any type of laboratory testing, not just to unit-use devices. Unfortunately, this type of detailed checking of individual error sources is quite inefficient and can become very time-consuming. One way to optimize the approach is to first assess the capability of statistical QC for detecting errors in many steps of the analytical process, then add just those checks needed for the steps not covered or tested by statistical QC. That's the basis of our Total QC strategy, which includes statistical QC, preventive maintenance, instrument function checks, method performance tests, and patient data QC [15]. This Total QC strategy can be extended for pre-analytical and post-analytical steps by adding specific monitors for potential error sources.
If you aren't old enough to recognize this title to a once popular song, you probably won't recognize that many of these new QC approaches have actually been around for a long time! They haven't been widely implemented because of their complexity and the need for computer tools and technology to support the planning and routine operation. There is greater potential for application today because of advancements in computer technology.
New computer planning tools. The planning of QC procedures can be quick and easy with appropriate computer tools, such as the QC Validator program [9,10]. The new version 3.0 of the QC Validator program will support 3 different QC designs, in agreement with Parvin's general recommendation for three-stage QC:
- a "startup" or "event-driven" QC procedure,
- a "monitor" or "non-event" QC procedure, and
- an Average of Normals (AoN) patient data QC procedure.
The QC planning process adheres to the NCCLS guidelines [2] and starts with the entry of the quality required for the test for your patients, accounts for the imprecision and inaccuracy observed for the method in your laboratory, and considers the number of control materials to be analyzed. Three different types of quality requirements can be used (allowable total error, clinical decision interval, European biologic goals). Multiple decision levels can be considered (up to four), along with the observed imprecision and inaccuracy at each of these levels. The selection of the QC procedures is made automatically on the basis of the criteria and logic built into the program. The default settings for the criteria and logic can be easily modified to accommodate preferences for different types of QC procedures for different marketplaces and for different QC software packages that may be available in an instrument, a stand-alone QC workstation, or a laboratory information system. An "automatic QC selection engine" can also be supplied that can be embedded in the QC software of an instrument, computer workstation, or laboratory information system.
New QC software and technology. A new generation of QC software is becoming available on instrument systems and PC workstations. The new QC programs offer more flexibility in the selection of control rules and the implementation of individualized QC designs on a test by test basis. They all support multirule applications and allow the user to select the combination of rules to be applied. To my knowledge, none of them support multistage designs for traditional statistical control rules, but there may be limited support for AoN calculations. More widespread use of patient data algorithms will depend on the capabilities of laboratory information systems. LIS's tend to be slow in upgrading their QC programs, thus the best chance for implementing these new QC approaches rests with the manufacturers of instrument systems who provide online QC programs and the suppliers of control materials who provide PC workstations.
Ultimately, laboratories need new QC technology - a totally automated QC process that provides automatic QC design, collection and interpretation of the QC data, automatic release of correct test results, identification and documentation of problems, support for trouble-shooting and corrective action, on-going data review and peer comparisons, and automatic adaptation or redesign where there are changes in method performance or the frequency of problems changes. Laboratories today often use 1st or 2nd generation QC technology with 4th and 5th generation measurement technology. QC has some catching up to do. These new QC approaches are long overdue.
While it is possible to support the planning and implementation of new approaches for quality control with new QC technology, the real issue today is whether laboratories are interested in quality control in an era when cost control has a higher priority. Bottom-line management has turned CLIA's minimum standards for quality into the maximum QC practices that are needed to satisfy inspection and accreditation. It appears that the laboratory inspector might be a more important customer than the patient today! Laboratories are doing what's needed to pass inspection, rather than what's needed to deliver quality services to the patient.
Quality management considers the patient's needs first and achieves cost reductions as an outcome of quality improvement. Improving laboratory QC can reduce costs by minimizing false rejections, reducing rework or repeat runs, and reducing the costs associated with the treatment of patients on the basis of erroneous test results. As Deming emphasized, improved quality will lead to improved productivity, which in turn will reduce costs and provide a competitive advantage [16]. That idea has been around nearly 20 years in American industry and almost 50 years in Japanese industry. But, it hasn't been taken seriously in healthcare organizations, in spite of all the rhetoric about quality assurance, total quality management, continuous quality improvement, and organizational process improvement. What's new is often old because it wasn't implemented earlier!
- U.S. Department of Health and Human Services. Medicare, Medicaid and CLIA Programs: Regulations implementing the Clinical Laboratory Improvement Amendments of 1988 (CLIA). Final rule. Fed Regist 1992;57:7002-7186.
- NCCLS. Statistical quality control for quantitative measurements: principles and definitions; Approved Guideline - Second Edition. NCCLS, 940 West Valley Road, Suite 1400, Wayne, PA, 1999.
- Eggert AA, Westgard JO, Barry PL, Emmerich KA. Implementation of a multirule, multistage quality control program in a clinical laboratory computer system. J of Med Systems 1987;11:391-411.
- Westgard JO, Groth T, deVerdier C-H. Principles for developing improved quality control procedures. Scan J clin Lab Invest 1984;44:19-41.
- Cembrowski GS, Chandler EP, Westgard JO. Assessment of "Average of Normals" quality control procedures and guidelines for implementation. Am J Clin Pathol 1984;81:492-499.
- Cembrowski GS, Westgard JO, Eggert AA, Toren EC. Trend detection in control data: optimization and interpretation of Trigg's technique for trend analysis. Clin Chem 1975;21:1396-1405.
- Westgard JO, Smith FA, Mountain PJ, Boss S. Design and assessment of average of normals (AON) patient data algorithms to maximize run lengths for automatic process control. Clin Chem 1996;42:1683-1688.
- Koch DD, Oryall JJ, Quam EF, Feldbruegge DH, Dowd DE, Barry PL, Westgard JO. Selection of medically useful quality-control procedures for individual tests done in a multitest analytical system. Clin Chem 1990;36:230-233.
- Westgard JO, Stein B, Westgard SA, Kennedy R. QC Validator 2.0: a computer program for automatic selection of statistical QC procedures in healthcare laboratories. Comput Method Program Biomed 1997;53:175-186.
- Westgard JO, Stein B. An automatic process for selecting statistical QC procedures to assure clinical or analytical quality requirements. Clin Chem 1997;43:400-403.
- National Cholesterol Education Program Standardization Panel. Current status of blood cholesterol measurement in clinical laboratories in the United States. Clin Chem 1988;34:193-201.
- Westgard JO, Hyltoft Petersen P, Wiebe DA. Laboratory process specifications for assuring quality in the U.S. National Cholesterol Education Program. Clin Chem 1991;37:656-661.
- Westgard JO, Wiebe DA. Cholesterol operational process specifications for assuring the quality required by CLIA proficiency testing. Clin Chem 1991;37:1938-1944.
- NCCLS. Quality management for unit-use testing; proposed guideline. NCCLS document EP18P. NCCLS, 940 West Valley Road, Suite 1400, Wayne, PA 19087, 1999.
- Westgard JO. Basic QC Practices. Appendix 4: Total QC Strategies. Westgard QC, Madison, WI, 1998, pp 229-237.
- Deming WE. Quality, productivity, and competitive position. Cambridge, MA., MIT Center for Advanced Study, 1982.
Past Essays:
|
|
