QP 4: Devising a Practical Process
The importance of implementing a quality-planning process is
evident from the general principles of Total Quality Management
(TQM) as embodied in the accreditation requirements of JCAHO's
IOP standards [1]. CLIA provides a focus on laboratory testing
processes and provides specific rules for validating the performance
of methods and planning QC procedures [2]. NCCLS addresses the
issue of how to perform quality-planning for QC procedures in
general terms [3]. However, none of these describe a detailed
process that can easily be implemented. Our objective in this
lesson is to devise a detailed step-by-step process that can be
supported by available tools or technology.
Review your intended applications
An understanding of your own uses of a quality-planning process
is important, particularly for assessing the practicality of different
tools and technology. If you are a laboratory quality coordinator,
your first interest may be in satisfying regulatory and accreditation
guidelines for planning your testing processes. A rigorous and
well-documented process will be important. If you are a manufacturer,
your interest may be in establishing performance specifications
for the precision and accuracy of new methods. If you are a laboratory
inspector or a technical field specialist for a manufacturer,
you may be visiting different laboratories and will need a "portable"
process that you can take with you - most likely installed on
your computer.
- Laboratory applications. Selecting control rules and
numbers of control measurements is, of course, an important application
in a service laboratory. In addition, the performance needed
by the method can also be determined if the QC procedures are
given, which should be useful in establishing purchase specifications
for methods, instruments, and systems. QC recommendations from
manufacturers and QC guidelines given in the literature can be
evaluated to be sure they are adequate for the quality required
for the test and the analytical performance claimed for the method.
It is also possible to compare allowable total errors, clinical
decision intervals, and biologic goals to determine which is
most demanding and should take priority in managing a testing
process.
- Manufacturers' applications. The design of new methods
and systems should be greatly aided by a quantitative approach
for setting performance specifications for imprecision and inaccuracy.
QC recommendations can objectively developed and validated based
on regulatory quality requirements, the design specifications
for imprecision and inaccuracy, and the common QC practices in
the marketplace. Customers can be supported and assisted in the
proper management of analytical systems by having a better understanding
of the relationships between the quality required for a test,
the imprecision and inaccuracy expected from a method, and the
QC procedures to be implemented.
- Regulatory and accreditation applications. It is interesting
that there is little documentation of the source and origin of
proficiency testing (PT) criteria, such as those allowable total
errors specified by CLIA. Regulatory agencies and PT providers
should evaluate the practicality of proposed PT criteria by comparison
with clinical decision intervals, taking into account the common
QC practices and expected method performance. Manufacturer's
QC product labeling should be reviewed to assess whether those
QC instructions are valid for the intended users. Laboratory
QC practices should be reviewed to assess whether they are valid
to assure the quality needed for the patient populations and
clinical applications of a healthcare organization. Laboratorians
and manufacturer's should be educated and supported to provide
more optimal management of the analytical quality of their tests
and systems.
Recognize two categories of applications
The main applications involve either
the (a) selection of the method of analysis or establishment of
performance specifications for imprecision and inaccuracy, or
(b) the selection of a QC procedure for a method in routine service.
In both cases, the first step will be to define the quality requirement
for the diagnostic test of interest. Then, as shown in the accompanying
figure, there are two variations of the planning process, depending
on whether the purpose is to select the method of analysis or
to select a QC procedure:
- To select a method of analysis or set performance specifications
for a method, the quality-planning process involves specifying
the QC procedure (statistical control rules, number of control
measurements or N) that will be employed and then setting the
developmental or purchase specifications for the imprecision
and inaccuracy of the method.
- To select a QC procedure for a method, the process involves
assessing method performance (imprecision and inaccuracy) and
then selecting the statistical control rules and number of control
measurements to be used.
The key step in both applications is the use of an appropriate
quality-planning tool that will translate the defined quality
requirement into specifications for the imprecision and inaccuracy
that are allowable and the QC that is necessary.
Identify a practical quality-planning
tool
A chart of operating specifications (or OPSpecs chart) is the
most practical tool because it provides all of the necessary information
on a single graph [4]. It is easy to use and easy to prepare using
a computer program, but it is complicated to understand. An analytical
quality-planning model is available to translate an allowable
total error requirement into the imprecision and inaccuracy that
are allowable and the QC that is necessary [5]. A clinical model
is available that accounts for pre-analytical factors, such as
within-subject biologic variation, as well as the analytical factors
- imprecision, inaccuracy, and QC [6]. The theory will be considered
later; our interest now is to demonstrate the practicality of
the OPSpecs chart for quality-planning applications.
|
An OPSpecs chart is like a map. You use it to find where you
are and to provide directions to where you are going. First you
need to get the right map, which requires defining the city,
county, or state of interest. This is analogous to defining the
quality requirement for a test. There are different maps for
different quality requirements, but all the maps are similar
in form. As illustrated in the accompanying figure, the map identifies
areas of deep water, shallow water, and solid ground. From a
quality-planning point of view, the objective is to be on solid
ground rather than ending up "in the drink." Given
the correct map, you can find your location by knowing the coordinates
on the x-axis and the y-axis, or you can look up the coordinates
to get to a location of interest. |
|
|
An actual OPSpecs chart will give the y-coordinate as the allowable
inaccuracy and the x-coordinate as the allowable imprecision,
as illustrated in the next figure. The area of "deep water"
is defined by a line that corresponds to a criterion for stable
method performance, most commonly a total error criterion composed
of the bias (method inaccuracy) plus two standard deviations
(method imprecision). This is a hazardous area and must be avoided
if you want to assure the quality required for a test. The solid
ground is delineated by a QC procedure, i.e., the limits of bias
and imprecision that are allowable for a specific control rule
or combination of rules and a given number of control measurements
(N). Several lines may be provided corresponding to several different
QC procedures. In the figure here, three different QC procedures
are depicted by the green, orange, and red lines. |
|
To select an appropriate QC procedure, find the location that
corresponds to the imprecision and inaccuracy observed for your
method. This is the "operating point" of the method
in your laboratory. Then, look to see if you're on solid ground
for any of the candidate QC procedures. In this example, only
the QC procedure corresponding to the green line could be used
to assure the quality of your test results. All that remains is
to look up the control rules and N that correspond to the green
line (usually be given by a "key" to the lines).
|
The next figure shows how an OPSpecs chart could be used to establish
performance specifications for the imprecision and inaccuracy
of a method. In this application, you select the QC procedure
to be used in routine operation and then determine the imprecision
and inaccuracy that are allowable. For example, if you pick the
orange line as illustrated here, then x-intercept of that line
will specify the maximum imprecision that would be allowable
if bias were zero. Note that the specifications for allowable
imprecision and inaccuracy are interdependent. Any allowable
bias above zero will reduce the imprecision that is allowable. |
|
Adapt the NCCLS QC planning process
Now that the general approach has been outlined and the application
of the OPSpecs chart has been illustrated, we need to develop
a more detailed step-by-step process. Given that the NCCLS guidelines
for planning QC procedures are the most well developed, they provide
a good starting point for devising a practical planning process.
Note that the NCCLS term "statistical QC strategy" is
replaced here with the term "QC procedure" to avoid
confusion with the "Total QC strategy" term used here
to describe an overall QC system that may include non-statistical
components (such as instrument function checks, patient data QC,
instrument maintenance, etc). The second important point is that
the NCCLS guidelines do not provide any "specifics"
for how to predict (step 4) and set goals for QC performance (step
5). In devising a step-by-step planning process here, QC performance
will be characterized by the probabilities of rejecting runs having
different sizes of errors, therefore there are two probabilities
that are of particular interest:
- Probability of false rejection, i.e., the chance of rejecting
a run when there are no errors except for the inherent random
error of the method;
- Probability for error detection, i.e., the probability or
chance of rejecting a run whether there is an error present in
addition to the inherent random error of the method.
Therefore, in devising a practical quality-planning process,
we will add some specifics to the NCCLS process, particularly
steps 3, 4, and 5.
Define a detailed step-by-step process
An eight-step quality-planning process is shown in the accompanying
flowchart. Here's a description of each of the steps:
Define the quality
required for the test. For practical purposes, it is easiest
to get started with requirements in the form of an allowable
total error, such as specified by proficiency testing or external
quality assessment programs.
- Assess method performance in terms of imprecision and
inaccuracy. Here's where method validation experiments are
important to provide the initial estimates of imprecision (from
a replication experiment) and inaccuracy or bias (from a comparison
of methods experiment). Later on, the estimates of imprecision
can be obtained from routine QC data and estimates of bias can
be obtained from monthly peer comparison data and proficiency
testing results.
- Assess QC performance of candidate procedures in terms
of the rejection characteristics or power curves. This information
is available in the scientific literature for most of the commonly
used QC procedures [7] and can be incorporated in quality-planning
tools and technology to facilitate the application [8].
- Utilize QC planning tools. The available tools include
QC simulation programs [9], power function graphs [7], critical-error
graphs [10], QC Selection Grids (QCSGs) [11], and OPSpecs charts
[4]. The OPSpecs chart is recommended here because of it is a
quantitative tool that is easy to use and readily available.
- Evaluate the probabilities of rejection for the operating
conditions in the laboratory. In the quality-planning process
recommended here, the probabilities for false rejection will
be minimized (below 0.05 or 5%) and error detection will be maximized
(0.90 or 90% and greater).
- Select appropriate control rules and the total number
of control measurements. A wide variety of control rules
are available. The rejection characteristics of each QC procedure
must be known if it is to be a candidate for implementation.
Candidate QC procedures include single-rules such as 12s,
12.5s, 13s, and 13.5s with Ns
of 2, 3, 4, and 6; multirules such as 13s/22s/R4s/41s/8x
with Ns of 2 and 4 and 13s/2of32s/R4s/31s/6x
with Ns of 3 and 6.
- Adopt a Total QC strategy that provides an appropriate
balance of statistical and non-statistical components. This
TQC strategy defines the relative amount of effort expended for
statistical QC, instrument function checks, method validation
tests, patient data QC, preventive maintenance, and operator
training. When HIGH error detection is obtained by statistical
QC (i.e., Ped of 0.90 or 90% detection of medically important
errors), the HIGH TQC strategy is to depend on statistical
QC and perform the minimum other QC required by regulations,
accreditation, and good practice guidelines. When MODERATE
error detection is obtained (Ped between 0.50 and 0.90), the
MODERATE TQC strategy is to balance the efforts over all
the QC components. When LOW error detection is available
(i.e., Ped less than 0.50), the LOW TQC strategy emphasizes
preventive measures because problems can not be detected by statistical
QC.
- Reassess the control rules, N, and TQC strategy when method
performance or quality requirements change. Given a quality-planning
process that is quick and easy to perform, it can be repeated
whenever changes occur or when methods are periodically reviewed.
Obtain the necessary tools or technology
A laboratory's ability to do anything efficiently often depends
on utilizing tools and technology to facilitate a process. Most
laboratory procedures have evolved from an initial qualitative
manual method (1st generation) that has then been systematized
and made more quantitative with tools such as diluters and photometers,
then automated through succeeding generations of technology until
complete systems are available that are highly efficient and productive
(such as todays 4th and 5th generation chemistry and hematology
analyzers). Quality planning, likewise, must evolve from a qualitative
manual method to a systematic process that utilizes standard tools
to a quantitative automated process that is quick and effective.
Concerning OPSpecs charts - the quality-planning tool recommended
here, different "generations" are available, as follows:
- Manual from scratch: using theoretical models available
in the scientific literature [5,6] with implementation via electronic
spreadsheets;
- Kit form: using preprinted charts in workbook form
(an atlas of maps), such as the OPSpecs Manual [12], or a standard
set of Normalized OPSpecs charts, which will be provided along
with these lessons;
- Semi-automated: using Internet calculation tools,
such as the "normalized" OPSpecs calculator; [see http://www.westgard.com/normcalc.htm]
- Automated: using a PC computer program - QC
Validator - that prepares OPSpecs charts for both analytical
total error requirements and clinical decision interval requirements
(version 1.1) and fully automates the selection of QC procedures
(version 2.0) [13,14].
- Highly automated: using a QC rule selection engine
that can be embedded in QC software to support the automatic
selection and design of QC procedures [see http://www.westgard.com/qcengine.html].
REFERENCES
- 2000-2001 Comprehensive Accreditation Manual for Pathology
and Clinical Laboratory Services. Joint Commission on Accreditation
of Healthcare Organizations, Oakbrook Terrace, IL, 1999.
- U.S. Department of Health and Human Services. Medicare, Medicaid
and CLIA programs; Regulations implementing the Clinical Laboratory
Improvement Amendments of 1988 (CLIA). Final rule. Fed Regist
1992;57:7002-186.
- C24-A2. Statistical Quality Control for Quantitative Measurements:
Principles and Definitions; Approved Guideline - Second Edition.
National Committee for Clinical Laboratory Standards, Wayne,
PA, 1999.
- Westgard JO. Charts of operating specifications (OPSpecs
charts) for assessing the precision, accuracy, and quality control
needed to satisfy proficiency testing criteria. Clin Chem 1992;38:1226-33.
- Westgard JO, Hyltoft Petersen P, Wiebe DA. Laboratory process
specifications for assuring quality in the U.S. National Cholesterol
Education Program. Clin Chem 1991;37:656-61.
- Westgard JO, Wiebe DA. Cholesterol operational process specifications
for assuring the quality required by CLIA proficiency testing.
Clin Chem 1991;37:1938-44.
- Westgard JO, Groth T. Power functions for statistical quality
control rules. Clin Chem 1979;25:863-869.
- Westgard JO. Assuring analytical quality through process
planning and quality control. Arch Pathol Lab Med 1992;116:765-9.
- Groth T, Falk H, Westgard JO. An interactive computer simulation
program for the design of statistical control procedures in clinical
chemistry. Comp Prog Biomedicine 1981;13:73-86.
- Koch DD, Oryall JJ, Quam EF, Feldbruegge DH, Dowd DE, Barry
PL, Westgard JO. Selection of medically useful QC procedures
for individual tests on a multi-test analytical system. Clin
Chem 1990;36:230-233.
- Westgard JO, Qulam EF, Barry PL. QC selection grids for planning
QC procedures. Clin Lab Science 1990;3:271-278.
- Westgard JO. Error budgets for quality management: Practical
tools for planning and assuring the analytical quality of laboratory
testing processes. Clin Lab Manag Review 1996;10:377-403.
- Westgard JO, Stein B, Westgard SA, Kennedy R. QC Validator
2.0: a computer program for automatic selection of statistical
QC procedures for applications in healthcare laboratories. Computer
Method Programs Biomed 1997;53:175-86.
- Westgard JO, Stein B. Automated selection of statistical
quality-control procedures to assure meeting clinical or analytical
quality requirements. Clin Chem 1997;43:400-3.
