Taking Autoverification to the Next Level – Is that up or down?
|
|
Taking Autoverification to the Next Level – Is that up or down?
|
|
January 1st, 2004
An updated version of this essay appears in the Nothing but the Truth about Quality manualJames O. Westgard, PhD, FACB
A couple of matters in the news caught my attention over the holidays. The November 2003 issue of Clinical Laboratory Science provided an interesting discussion of “autoverification”, which was defined as a “set of logic or computer rules that are designed to mimic exactly what laboratorians would do it they were manually verifying test results – without errors” [1]. Dr. Larry Crolla and I just published an article on the “Evaluation of Rule-based Autoverification Protocols” in the September/October 2003 issue of Clinical Laboratory Management Review [2]. You can imagine my interest in looking at the rules being used in real laboratory operation vs the evaluation guidelines we have developed.
Another matter that has achieved particular attention in the Wisconsin newspapers is the mutual fund scandal. Strong Financial happens to be a Wisconsin corporation that has implicated in this scandal. Strong holds the contract the State of Wisconsin “Edvest” business, which is the official state program that allows parents to invest money for their children’s college education. As you may know, a number of mutual funds have been implicated in after hours trading, something that isn’t necessarily illegal, but is still considered to be wrong. The December 5, 2003, Capital Times (a Madison newspaper) described the issue this way:
“Investigators looking at the mutual fun industry are focusing on so-called market-timing transactions, in which short-term, in-and-out trades are used to capitalize on market-moving news. The process is not illegal, but many fund companies – including Strong – have policies against it because it increases costs and hurts long-term shareholders. Regulators have indicated that it is fraudulent for a fund to allow selective market-timing without disclosing that to shareholders.”
You may be wondering why I mention these seemingly unrelated topics here. Actually, there is a relationship – what’s legal is not the same as what’s right!
The autoverification process being discussed in the Clin Lab News article failed to include a basic rule that internal QC must be part of the ongoing validation of test results. It appears that a common mode of operation in laboratories today is to setup the analytical system, calibrate it, run fresh controls, rerun controls as necessary, and then do whatever else that is necessary to force the control results to be in. Once that’s accomplished, they proceed to autoverify for the rest of the day.
I hope that practice doesn’t represent “exactly what laboratorians would do if they were manually verifying test results – without errors.” However, manufacturers tell me that is exactly what laboratories are doing and exactly how laboratories want autoverification to function. Even if the manufacturers provide the capability for ongoing QC monitoring, laboratories don’t or won’t use that capability.
In my opinion, implementation of autoverification in this manner is a step in the wrong direction, a level down rather than up! And, under CLIA, that practice is legal because compliance only requires running 2 controls every 24 hours! If your only interest is to be in compliance with CLIA, you’re good to go. Just be sure to advertise your laboratory as a minimum compliance business.
The particular rules being included in the autoverification protocols being discussed in the Clin Lab News article are mainly of the following type:
- Is the test result within the normal range, or a similarly defined interval where interpretation is not expected to be critical?
- Is the test result within a specified range for a delta check?
- Is there any inconsistency between this test result and another test result?
The quality of test results in the normal range does matter! Look at what’s happening with the new ADA guidelines. The latest suggestion is to drop the glucose cutoff even further, to 100 mg/dL. That’s within the expected reference range for glucose. Even normal results need to be correct given today’s aggressive interpretation of laboratory tests.
Patient data checking rules are not as powerful as statistical QC rules! These data checking rules are logical, but they’re not very sensitive for detecting errors. There certainly is anecdotal information to justify these kinds of rules, but they generally haven’t been carefully evaluated in a manner that permits comparison to statistical QC. Work that I’ve been involved with on patient data algorithms has shown the need to use a group of patient samples to provide reasonable error detection. For example, the mean anion gap for a group of 8 patients is much better for monitoring test performance than individual anion gaps [3]. It is also much better to use of an “average of normals” (AoN) for a group of patient results, rather than compare individual patient results to a reference range. AoN algorithms, in the best case application, require a minimum of 10 to 20 samples, but more often will require a hundred or more [4,5].
Lab-specific autoverification rules have not been well studied! The most extensive application of autoverification is found in Europe, where the approach is termed “medical validation.” Two programs, VALAB and LabRespond, have been studied and their performance documented in the literature [6,7]. These programs employ a series of rules that have been developed and tested by professional groups, therefore they represent the collective wisdom of many laboratories, rather than rules selected by an individual laboratory, which is the practice emerging in the US. It should also be noted that statistical QC is part of the overall verification process, e.g., the third level of checking in LabRespond, prior to checking by patient data algorithms.
While I agree that patient data checking rules will catch some gross errors, I don’t think such rules by themselves can assure the quality required for many of the medical decisions for which the tests are being used today. Laboratories still need to utilize statistical QC procedures that are designed for the quality required by the test and the precision and accuracy observed for the method. In this age of evidence based medicine, the existing evidence will only justify an autoverification process that incorporates statistical QC.
But controls are already being run! Actually, just running controls is not enough. The laboratory needs to be sure that the right control rules and right number of control measurements are being utilized. A simple methodology for doing this makes use of the sigma-metric for the method, which can be calculated from the quality required for the test (TEa, allowable total error) and the precision (s) and accuracy (bias) observed for the method [Sigma = (TEa – bias)/s]. The QC rules and number of control measurements can be selected to fit the method’s sigma-metric, as follows:
- For a 6 sigma process (or higher), use 3.5 SD control limits with N=2;
- For a 5 sigma process, use 3.0 SD control limits with N=2;
- For a 4 sigma process, use 2.5 SD control limits or a multirule procedure with N=4;
- For a 3 sigma process, use a multirule procedure with N of 6 or 8.
- For less than 3 sigma, method performance must be improved before the method can be used for routine production.
It costs too much to do more QC! Given that autoverification is being recommended primarily for highly automated chemistry and hematology analyzers that can produce hundreds and thousands of test results in a 24 hour period, no one can argue that cost of doing QC is an issue in these applications. Laboratories can and should be willing to invest a couple percent of the operating costs in QC, say an additional 1, 2, or 3 controls per every 100 patient tests. Given the small incremental cost of additional tests on these high production analyzers, this level of cost for additional QC is really not an issue. And if test quality is so good that they aren’t needed, the laboratory will at least have the documentation to prove it.
Rules are rules! These data checking rules are useful as a complement to statistical QC, but they’re not a substitute for statistical QC. Some rules are more important than others, in particular, a rule to include ongoing statistical QC is essential. That was the issue that stimulated our paper in CLMR [2]. Using the analogy of a golf game for evaluating autoverification rules, we developed a scorecard similar to the numbers you would expect to see in a golf game. We also imposed a condition that there was only one golf ball and if the player encountered severe hazards and lost the ball, then the game was over, meaning the autoverification protocol was a loser. The lack of an integral QC procedure as part of autoverification is such a hazard. It is dangerous to autoverify patient results if there is no ongoing verification of QC results.
In the exposure of recent business fraud, the leadership has NOT come from the US national government and its various agencies and commissions that are charged with assuring compliance with the law. Instead, Eliot Spitzer, the New York State Attorney General, has led the way in exposing fraud, malpractice, and criminal activities by many corporations, accounting firms, and their executive officers.
My point is that we cannot look to CMS to assure that laboratories do what is right as well as legal. It will be up to the states and the deemed providers (CAP, JCAHO, COLA, etc) who inspect laboratories. They will need to impose standards of practice that are higher than CLIA - standards that consider what is right and effective for assuring quality in healthcare and healthcare laboratories. Given that most states have severe financial difficulties right now, professional deemed providers are the ones who have the greatest opportunity to step up and fill the void. That’s actually the best way to get quality back on the agenda of the laboratories and also to drive improvements in quality by manufacturers. We, as customers, can demand higher quality and should select those vendors who adhere to higher quality standards.
Professional standards must once again take priority to assure quality of testing in healthcare laboratories. In the last decade, we have learned first hand the effect of government regulations on the laboratory and no one should be content with adhering to the minimum standards that are encoded in the Final CLIA Rule. We can and should take back our responsibility for quality and we must do better.
Autoverification is just one example where the professional responsibilities in the laboratory are the major issue for assuring quality test results for our patients. If quality depends on doing the right things right, then the implementation of any autoverification process requires selecting the right rules and implementing them in the right way. Integrated and on-going QC checking should be one of the rules in any autoverification process. The statistical QC procedures should be properly designed for the test and method and properly implemented in the software, regardless whether that software is part of the instrument, middleware, or a laboratory information system.
- Auxter-Parham, S. Taking autoverification to the next level: New tools make it easier to increase efficiency. Clin Lab News 2003; 29(Number 11, November):1,6-7.
- Crolla LJ, Westgard JO. Evaluation of rule-based autoverification protocols. CLMR 2003;17:268-272.
- Cembrowski GS, Westgard JO, Kurtycz DFI. Use of anion gap for the quality control of electrolyte analyzers. Am J Clin Pathol 1983;79:688-696.
- Cembrowski GS, Changler EP, Westgard JO. Assessment of “Average of Normals” quality control procedures and guidelines for implementation. Am J Clin Pathol 1984;81:492-499.
- Westgard JO, Smith FA, Mountain PJ, Boss S. Design and assessment of average of normals (AON) patient data algorithms to maximum run lengths for automatic process control. Clin Chem 1996;42:1683-1688.
- Valdiguie PM, Rogari E, Philippe H. VALAB: expert system for validation of biochemical data. Clin Chem 1992;38:83-87.
- Oosterhuis WP, Ulenkate HJLM, Goldschmit HMJ. Evaluation of LabRespond, a new automated validation system for clinical test results. Clin Chem 2000;46:1811-1817.
Other Essays:
