Quality Requirements and Standards
The Elephant in the Laboratory
As with January of every year, there’s an impulse to set goals. Perhaps a worthy goal of 2014 would be to choose the right goals for quality.
2014: The Elephant in the Laboratory
Sten Westgard, MS
January 2014
There’s an elephant in the laboratory, and its name is quality. While 2014 promises to bring a wave of changes, particularly to healthcare laboratories in the US, very few people are talking about an idea that is taken for granted: the quality of our testing.
The Affordable Care Act, for all its laudable intentions, will manifest itself to the laboratory as a cleaver. Cuts in reimbursement rates are continuing, deepening, and the recent budget “deal” provided no relief. The looming challenge is that the cost of testing will be bundled into other diagnosis related group (DRG) codes, removing direct payments to labs and making them fight for their share of a single payment to the whole health system.
As laboratories face these greater cost pressures, how will they address quality? Will they ignore it? Will they pretend that quality is never going to be threatened by all of this budget cutting? Will they assume that all manufacturers provide the same level of quality in their test method?
As the process of producing laboratory results and generating medical diagnoses is streamlined, automated, and more and more tightly coupled, the quality of testing will actually matter more. Clinicians are going to have less ability to “interpret” results, or order repeats, and are more likely to have decisions popped up at them by “electronic triggers” and medical “decision support” systems. Thus, the rule from the earliest age of computer programming still remains true: Garbage In, Garbage Out. Despite all the innovation and advances, if at the core, our laboratory tests are not providing the right level of quality, we will not be helping patients or making care more efficient, we’ll just be mistreating them faster than ever.
An elephant of a different color: Our quality goals don’t agree
Q: How do you shoot a blue elephant?
A: With a blue elephant gun?
Q: How do you shoot a red elephant?
A: Hold his trunk until he turns blue, then shoot him with a blue elephant gun
Q: How do you shoot a purple elephant?
A: Paint him red, hold his trunk until he turns blue, and then shoot him with a blue elephant gun
As far back as 1999, an international consensus identified the many different sources of quality requirements and arrange them into a hierarchy. We’ve come to call it the Stockholm Consensus Hierarchy, and it places more evidence-based quality goals in higher regard than goals set by consensus.
While this conference established a preference for which goals to use, there was never any movement toward standardization or harmonization of the use of quality goals in laboratories worldwide. In the US, labs continue to use CLIA goals, in Germany, the Rilibak rules are dominant, while in the rest of Europe, I would estimate that the biologic variation-based goals (sometimes called “Ricos goals”) are more popular, and in Australia, the (Royal College of Austral-Asia) RCPA has its own well-respected set of Allowable Limits of Performance.
This multiplicity of quality goals means it’s harder for labs to understand the performance of methods. If a laboratory in Malaysia publishes a paper indicating they are achieving Six Sigma performance with their assays, one needs to read the details to understand which goals are being used first.
We even seem to be experiencing not a gradual harmonization quality goals, but the reverse: a fragmentation of goals. There are well-meaning scientists who would prefer to abandon the Allowable Total Error approach and return to the world of forty years ago, when imprecision and bias were considered separately, and never take into consideration the combined effect of these two types of error. But like two credit cards linked to the same account, you can’t pretend that running up the limit on one card doesn’t impact what you can do with the other card. Going further still, there are those who want to embrace a new age of reporting uncertainty instead of quantifying quality. There are those who want to include, or expand, quality goals, so they encompass not only analytical error, but also pre-analytical error and post-analytical error, and possibly, pre-pre-analytical error and post-post analytical error. In the US, quality requirements may even be supplanted by Risk Plans, an amorphous collection of guesstimates, gut feelings, and gross oversimplifications that will masquerade as quality.
Unfortunately, it doesn’t look like quality requirements and goals are about to enter an era of consolidation. Instead, they may proliferate. We may have so many more choices about quality that we suffer decision fatigue and stop trying to figure out how good a test needs to be.
The Blind Men and the elephant: The goals of our quality goals probably don’t agree
Another famous elephant analogy comes to mind, this one of the blind men and the elephant. The story goes, a group of blind men encounter an elephant and each one touches a different part of the animal. One man touches the tusk and thinks the creature is shaped like a solid pipe. One man touches a leg and thinks the creature is shaped like a pillar. One man touches the trunk and thinks the creature is shaped like a tree branch. Another touches the tail and thinks the creature is shaped like a rope. Of course the answer is that all of those perspectives are correct, they’re just not seeing the total picture.
One of the reasons that many of our quality requirements don’t agree is that they come not only out of different groups, but also out of different thinking. Some of these goals are formulated to specify just the maximum allowable imprecision and maximum allowable bias. Other goals are from a time so long ago that they were specifying performance acceptance for methods that were three generations ago in terms of technology.
Another concept to consider is that some of these goals are only specifying “Two Sigma” or “Three Sigma” levels of success. That is, the goals is described but the rate of success isn’t. When we define a goal and demand that the goal be hit or achieved 99.99999% of the time, we are describing a Six Sigma goal. When we define a goal but only demand 95% success, we are now describing a 3.2 Sigma goal (on the short-term Sigma scale). When we define a goal for only 80% success (which is typical of PT and EQA programs), do we realize we may only be specifying a 2.4 Sigma goal?
With that in mind, we may actually have many goals that describe the same level of performance. A method that achieves only 3 Sigma when using a Ricos goal may achieve a far higher Sigma-metric when using a CLIA goal. But the intention of the Ricos goal and the CLIA goal may be different. We may expect a high level of success when using a “wider” goal such as CLIA, but allow for more deviation when using a demanding goal such as those found in Ricos. You can see illustrations of this in our series of applications comparing quality goals.
To see a very current example of a more specific form of quality goals, one need only look at the FDA draft guidance for Blood Glucose Meters released in early January 2014:
“[Y]ou should demonstrate that 99% of all values are within ± 10% of the reference method for glucose concentrations > 70 mg/dL, and within ± 7 mg/dL at glucose concentrations < 70 mg/dL. To avoid critical patient management errors, no individual result should exceed ± 20% of the reference method for samples >70 mg/dL or ± 15 mg/dL <70 mg/dL.”
Here we see two goals, the first that places a 99% success rate (a 3.9 Sigma specification on the short-term scale) at 10%, and the second that places a Zero defect rate (a 6 Sigma specification on the short-term scale) at 20%.
If we just take into account the imprecision of the method, and work on the long-term Sigma scale, we can calculate that in order to achieve these goals, we need in the former case, a CV of 2.5%, and in the latter case, a CV of 3.33%. It turns out the 10% goal is more demanding than the 20% goal; if we can meet the first goal, we’ll also be able to meet the second. Of course, if bias is present (and really we can't pretend it's not there), then precision demands are even higher.
To express a quality goal with an expected success rate is something that is quite common when describing error grids (Dr. Jan Krouwer has much discussion on error grids on his blog). Either we need to begin asking our regulators and accreditation providers to specify success rates in the goals they set forth, or we must begin assigning a success rate to those goals ourselves. Our long-standing practice has been to assume that for any goal stated, we should try to achieve a Six Sigma rate of performance in achieving that goal. For CLIA goals and some other sources (Rilibak, the Minimum EQA goals from Spain), that may be appropriate, but there may be a need to debate the success rate implied or expected of such goals as Ricos, RCPA, etc.
What will you do?
Q: How do you know an elephant is in the bathtub with you?
A: You can smell the peanuts on its breath
Q: How can you tell an elephant has been in your freezer?
A: You can see the footprints in the ice cream.
Q: What time is it when an elephant sits on your fence?
A: Time to build a new fence.
Laboratories have a choice: they can continue to ignore the elephant in the room, and instead adopt a a mask of ignorance. But gradually, as Orwell puts it, for those who wear a mask find their “face grows to fit it.” Pretending to ignore the problem eventually results in real ignorance of the true analytical quality being delivered to patients.
It’s time to mend fences, acknowledge our problems, and confront the challenge of choosing the right goal, with the right success rate, implementing with the right QC, and ultimately finding the right way to deliver quality patient results.