QP-6: Adopting the OPSpecs Chart
as Your Planning Tool
|
|
So far, these lessons have focused on the need for quality-planning,
based on the lack of independence in past laboratory practices
for analytical quality management, the emerging principles of
Total Quality Management, and the existing regulatory, accreditation,
and QC practice guidelines. A step-by-step quality-planning process
has been described drawing on the NCCLS QC practice guidelines.
The first step of this process requires definition of the quality
required for a test, which can be stated in several different
formats - all of which are important in a system of quality standards.
The bottom line in the laboratory is knowledge of the operating
specifications, which are the imprecision and inaccuracy allowable
for a method and the QC needed to monitor method performance and
assure the desired quality is achieved.
Operating specifications are different from quality goals or
quality standards because they consider QC as an integral part
for monitoring the instability of a method, whereas most quality
standards generally assume that method performance is stable and
therefore don't consider QC. In the lingo of today, this is a
paradigm shift - a new perspective or a different view of the
situation. QC needs to be designed into the testing process, in
addition to analytical imprecision and inaccuracy.
This new perspective provides a more complete and comprehensive
view of what is necessary to manage the analytical quality of
a laboratory testing process. It's also more complicated because
of the interdependence of imprecision, inaccuracy, and QC. The
better the imprecision and inaccuracy, the easier it is to QC
the process. The worse the imprecision or inaccuracy, the more
difficult it becomes to adequately QC the process. The interactions
of these three critical factors must be considered.
The best tool for showing these interactions is the chart of
operating specifications, or OPSpecs chart. The concept of the
OPSpecs chart has been introduced, using the idea of a map that
will help you get to solid ground. Example applications of OPSpecs
charts have been illustrated for a cholesterol test that has well-defined
standards of quality in US national regulations (CLIA-88) and
clinical practice guidelines (NCEP). Now is the time to learn
the details of how to use OPSpecs charts. For the theory of why
it works, see references 1 and 2.
How to read an OPSpecs Chart
An OPSpecs chart contains information about the type of quality
requirement, actual quality desired, the imprecision and inaccuracy
allowable for different QC procedures, details about the control
rules and number of control measurements, and information about
the error detection and false rejection characteristics of the
QC procedures. See the figure on the "ABCs of reading an
OPSpecs Chart" for guidance on finding all this information.
A. Start with the
title that appears at the top of the chart. The title identifies
the following:
- Type of quality requirement, TEa for an allowable total error
or Dint for a clinical decision interval. In this example, the
OPSpecs chart was prepared for an analytical total error requirement.
To prepare a chart for a clinical decision interval requirement,
you will need an electronic spreadsheet or computer program,
such as QC Validator.
- Desired quality, in % of a medically important decision level,
e.g., 10% for cholesterol in this example.
- Error detection, in %, such as 90% in this example, which
is the same as a probability of error detection (Ped) of 0.90.
The AQA(SE) stands for Analytical Quality Assurance for Systematic
Error. Charts are also available for 50% AQA(SE). With the aid
of the QC Validator computer program, charts can also be prepared
for 25% AQA and also for random error (RE).
B. Look at the axes of the chart.
- The y-axis shows allowable inaccuracy, or the method bias
in units of %, i.e., relative to the medical decision level of
interest.
- The x-axis shows allowable imprecision, or the standard deviation
(s) in units of %, which is the same as the coefficient of variation
or CV.
|
|
- The actual performance of a method can be displayed by plotting
an"operating point", as shown in the middle of this
chart (y-coordinate of 3.0% and x-coordinate of 3.0%).
|
|
C. Inspect the lines that describe the limits
of allowable bias and allowable imprecision, i.e., define the
solid ground.
- The partially hidden line shows the limits of stable performance
that correspond to a total error criterion of bias + 2s, which
is commonly used as a criterion for acceptable performance in
the initial evaluation or validation of a method. Remember that
this criterion assumes stable method performance and does not
consider the need for QC - that's why this line is the highest.
- The other lines correspond to certain control rules and numbers
of control measurements, which are identified in the key at the
right side of the chart. Follow the arrows across to find the
details of individual QC procedures. Note that the order of the
lines - top to bottom on the chart - corresponds to the order
of the lines top to bottom in the key at the right.
D. Match the lines
in the key of the graph to get the details about each QC procedure.
- The dots and dashes in the lines on the graph match the dots
and dashes in the lines in the key. For example, the solid line
on the graph corresponds to the solid line in the key. Often
it is easier to match up the lines based on their top-to-bottom
order on the graph and in the key.
- The control rule or rules are shown in the first column of
the key. The abbreviations are of the form AL, where
A is the symbol for a rule or the number of control measurements
and L is the control limit. For example, the top line is for
a 12s rule and indicates a run is to be rejected when
1 control measurement exceeds a 2s control limit. This corresponds
to a Levey-Jennings control chart having control limits set as
the mean plus/minus 2 standard deviations.
- The probability of false rejection, Pfr, is listed
in the second column. Ideally, this figure should be less than
0.05, or 5%, to minimize the false alarms from the QC procedure.
Note that for the 12s rule with N=2, the probability
for false rejection (Pfr) is 0.10, which means a 10%
chance of falsely rejecting each run. The use of 2s control limits
with N=2 should be generally discouraged because they cause a
10% waste in the production of a testing process, even when the
process is working perfectly. It gets even worse for higher Ns
(14% for N=3, 18% for N=4).
- N is the total number of control measurements per analytical
run. An N of 2 could refer to 2 measurements on a single control
material or 1 measure ment on each of two different materials.
Remember that N is the "total number" of control measurements
that can be spread over different control materials being used.
- R is the number of runs over which the control rules are
applied. In this example, all the rules are applied only in the
1st run. However, if a 41s rule were added to the
multirule procedure, the rule could only be applied if there
were 2 runs each having 2 control measurements to give the total
of 4 needed to apply the rule. Likewise, if a 10x
rule were added, there would need to be 5 runs to accumulate
the 10 measurements.
How to determine method performance
specifications
Recall the steps in the planning process when the intent is
to determine the imprecision and inaccuracy that is needed by
a method:
- Define the quality required for the test, e.g., a 20% clinical
decision interval for cholesterol on basis of NCEP treatment
guidelines.
- Obtain the OPSpecs chart for a 20% Dint, as shown
in the accompanying figure. Note the title of the OPSpecs chart
states Dint 20% with 90% AQA(SE), indicating it has been prepared
for a clinical quality requirement of 20% and 90% detection of
medically important systematic errors.
- Specify the laboratory's preferred QC procedure, e.g., 1
2.5s with N=2 as shown by the solid line. Note that the
12s procedure with N=2 is being avoided here because
of its high false rejection rate of 10%.
- Read the x-intercept for the solid line that corresponds
to the preferred QC procedure, in this case the line for 12.5s
with N=2. In this example, it looks like a method CV of 2.7-2.8%
is needed if method bias is zero. If method bias were 1.0%, then
a CV of about 2.5% would be needed; if method bias were 3.0%,
then a CV of 1.9% would be needed.
How to select a QC procedure
Again, let's go through this application step-by-step to illustrate
how OPSpecs charts are used:
- Define the quality required for the test, in this case, an
allowable total error of 10% according to the CLIA criterion
for acceptable performance for a cholesterol test.
- Obtain the OPSpecs chart for 10% TEa, as shown in the accompanying
figure. Again, start by reading the title of the chart which
indicates TEa of 10% for 90% AQA(SE). Note that this chart is
for N=2 QC procedures which are preferred to keep the cost of
QC low.
- Plot the operating point for the method, i.e., the observed
standard deviation or CV (in %) is the x-coordinate and the observed
bias (in %) is the y-coordinate.
- Inspect the lines showing the allowable limits of imprecision
and inaccuracy for different QC procedures. Select a line above
the operating point. If none is available, as in this example,
it will be necessary to consider other QC procedures having higher
Ns or to settle for lower error detection, perhaps 50% AQA(SE).
- Utilize other OPSpecs charts if necessary. If selecting a
QC procedure to work with 2 control materials, the strategy is
to inspect OPSpecs charts in the following order:
- N=2 with 90% AQA
- N=4 with 90% AQA
- N=4 with 50% AQA
- If using 3 control materials, the strategy is to inspect
OPSpecs charts in the following order:
- N=3 with 90% AQA
- N=6 with 90% AQA
- N=6 with 50% AQA
- The objective is to select a QC procedure having 90% error
detection with 5% or less false rejection and the lowest number
of control measurements possible. If necessary, settle for 50%
error detection and compensate by beefing up other non-statistical
control procedures as part of the overall or Total QC strategy.
In the example here, it's not possible to achieve 50% AQA even
with multirule QC procedures and N up to 6, as shown by the OPSpecs
chart titled "50% AQA(SE)". The operating point for
a 3% CV and 3% bias is still above all the lines for the QC procedures
given in the key here, which now include single-rule and multi-rule
procedures with Ns of 2, 4, and 6.
- Formulate a Total QC strategy that provides an appropriate
balance of statistical and non-statistical procedures (such as
preventive maintenance, instrument function checks, method validation
tests, patient data QC, in-service training, staffing with operators
who have maximum experience and minimal rotations). This TQC
strategy can depend primarily on statistical QC when a solution
is obtained from a 90% AQA chart. A balanced strategy is needed
when the QC selection is made using a 50% AQA chart. If less
than 50% AQA, the maximum statistical QC that is practical should
be selected, but in addition, efforts should be made to maximize
other non-statistical components and to improve the performance
of the method. It may even to advisable to acquire a new method
having better imprecision and lower bias, rather than expend
so much effort trying to control a method that does not provide
the necessary analytical performance.
How to assess need for quality improvement
Another useful application
of the OPSpecs chart is to assess the improvements in analytical
quality that are needed to simplify QC and reduce the number of
control measurements. For example, the cholesterol application
considered above is shown on the accompanying OPSpecs chart for
a TEa of 10%, 90% AQA(SE), and QC procedures with Ns from 2 to
6.
- The effect of reducing method bias is shown by the vertical
arrow located at 3.0% on the x-axis. If method bias were reduced
to zero, then a method with a CV of 3.0% could be controlled
using a multirule QC procedure with an N of 6. That's a costly
QC procedure, but at least the necessary analytical quality would
be assured.
- The additional benefit of improving method imprecision is
shown by the dotted horizontal arrow. If the method CV were improved
to 2.5%, adequate control is possible with single-rule or multirule
QC procedures having Ns of 4. If the method CV were improved
to 2.0%, QC procedures with Ns of 2 would be adequate.
The ability to assess the benefits of improvements in method
performance is one of OPSpecs chart's real advantages. Of course,
this cycles back to its earlier application for setting performance
specifications. If the necessary analytical performance were achieved
initially through careful selection and evaluation of the method,
then QC will turn out to be simple and easy to perform. However,
when the QC selection application demonstrates costly QC with
high N, you have the information to help you assess the benefits
form any improvements in method performance.
Note also that the demands of different quality requirements
can be compared by having OPSpecs charts for both the clinical
requirement (1st figure, NCEP clinical decision interval of 20%)
and the analytical quality requirement (last figure, CLIA allowable
total error of 10%). As shown earlier, to achieve the quality
needed for a cholesterol test that will be interpreted according
to the NCEP treatment guidelines, a method should be selected
that has a CV of 2.7% or less (when bias is 0.0 and the laboratory
intends to monitor performance with only 2 control measurements
per run). Compare this with the demands of the CLIA proficiency
testing requirement where the method needs a CV of 2.0% or better
(when bias is 0.0). The clinical requirement for patient treatment
is less demanding than the regulatory requirement for method performance.
It makes no sense to have a more demanding requirement for
analyzing proficiency testing samples than for analyzing patient
samples! This inconsistency is due to a lack of understanding
of the nature of these quality requirements and the inherent difficulty
of comparing apples and oranges. However, the OPSpecs methodology
can translate both requirements into equivalent terms (operating
specifications) that can be compared - another advantage of the
OPSpecs tool.
REFERENCES
- Westgard JO. Assuring analytical quality through process
planning and quality control. Arch Path Lab Med 1992;116:765-769.
- Westgard JO. Charts of operational process specifications
("OPSpecs Charts") for assessing the precision, accuracy,
and quality control needed to satisfy proficiency testing performance
criteria. Clin Chem 1992;38:1226-1233.
