Quality Standards

To study the usefulness of the calculations from The Quality of Laboratory Testing series, we took a look at one proficiency testing program, the New York State Wadsworth program. Rather than look at a single event, we looked at all the events for two years.

Looking at the Proficiency Testing Data

April 2006

[Note: This QC application is an extension of the lesson From Method Validation to Six Sigma: Translating Method Performance Claims into Sigma Metrics. This article assumes that you have read that lesson first, and that you are also familiar with the concepts of QC Design, Method Validation, and Six Sigma. If you aren't, follow the links provided.]

We have previously presented data and calculations about the Quality of Laboratory testing for several touchstone tests. The conclusions of the studies are in marked contrast with the conventional wisdom. Where the laboratory professionals usually assume that test performance is more than adequate than needed for medical usefulness, the data tell the opposite story: many test methods are performing poorly, and some are performing far worse than needed for proper medical use.

The methodology of the studies have been previously presented here. To summarize it briefly, we selected data from individual proficiency testing events from all the proficiency testing programs with the publicly available data.

To study the usefulness of these calculations, we took a look at one proficiency testing program, the New York State Wadsworth program. Rather than look at a single event, we looked at all the events for two years.

Broad measures of Quality from PT data

In addition to the three summary metrics (NTQ, LMQ, and NMQ), we have calculated the Weighted Average CV, eighted Average Bias, and the Sigma calculated by those weighted average figures.

Of these measures, it's clear that the most volatile metrics are the National Test Quality and the Weighted Average Bias. The bias changes most from event to event. The NTQ takes the bias into account in an unweighted form. The most stable measures are the Weighted Average CV, and the Local Method Quality (which is based solely on the imprecision measurement).

What do these metrics mean in practice? Using our previous "rules of thumb" on Sigma metrics, we can determine roughly what control rules and N would be required for each event.

As you can see, with the metrics that include bias, they are switching between N=4 QC procedures to Maximum QC procedures (basically, using as many controls and rules as possible). With NMQ, there is only one switch, when the sigma reaches above 5.0, which allows the user to reduce QC to wide rules with N=2.

Next, with the weighted CV and bias figures, we can actually calculate the specific rule recommendations with EZ Rules:

You can see the six events represented visually here:

But what if all these laboratories were using the CLIA minimum of 2 controls with limits set at the traditional 2s? What error detection would they by acheiving?

Notice the big difference in error detection capability. Instead of catching errors in the first run where they occur, it might take 2 runs. And remember that when using 2s limits and 2 controls, you can expect up to 10% false rejections. So you should have at least 3 rejections in a month (if you have one run each day). If you're not, that means your limits or your sd are set wider, larger than they should be - and the error detection figures here are even lower.

Data on specific instrument groups

Looking at specific instruments, even more interesting details become clearer.

Because of the differing numbers of instruments in the survey, it's useful only to compare like with like. For instance, comparing the Abbott Aeroset with the Dade Dimension based on this data probably isn't as reliable as comparing the Abbot Aeroset with the Roche Integra. Below, you can see the results of each PT event at the critical level of interest, as well as the resulting average as a solid "operating point":

Here's a visual display of the three instruments in the middle of the pack:

Finally, here is a comparison of the last three instruments.

What have we learned?

The data in the proficiency testing surveys is fairly stable. Taking a fine grained approach, the data is even more stable in differentiating between the performance of instruments.