Thinking Fast and Slow about Risk QC
As 2013 draws to a close (boy, it went fast!), we take some time to think about one of the biggest changes in laboratory quality control (at least for the US): Risk QC. Is Risk QC a quick and easy solution to all our problems? Not so fast...
Thinking Fast and Slow about Risk-Based QC Plans
James O. Westgard, PhD
DECEMBER 2013
Daniel Kahneman’s book “Thinking, Fast and Slow” provides many interesting insights into people’s thought processes and decision making. Kahneman is a Nobel Prize winner in Economics, but much of his work is about the psychology of thinking. “Thinking fast” involves intuition and he calls it System 1. “Thinking slow” is called System 2 and it involves logical reasoning and statistical thinking. We switch from one mode to the other, depending on the question or issue that needs to be addressed. The most common mode of operation is “thinking fast” which provides an immediate and ongoing assessment of our current situation. In contrast, “thinking slow” often requires effort and discipline.
As an example, what are the answers to the following questions?
- 4*5=?
- 6*7*8=?
The answer to 4*5 comes quickly to our minds (=20). The answer to 6*7*8 requires mental effort and a deliberate process, possibly even requiring pencil and paper (=336). In fact, when we first see 6*7*8 we may even mentally avoid answering the question because it requires additional effort. Kahneman suggests that our “thinking slow” process is often “lazy” and requires deliberate efforts for engagement.
In our current laboratory world where risk assessment is being introduced to develop Quality Control Plans, Kahneman’s modalities of thinking provide some interesting insights.
Consider the choice of 2-factor vs 3-factor risk models. The CLSI EP23A guideline [2] recommends that laboratories employ a 2-factor model rather than the standard 3-factor model that is common in most risk assessment applications. Individual factors are ranked, then combined to assess risk. From the simple example above where 2 or 3 numbers are combined, it is apparent that a 2-factor model lends itself to an intuitive assessment, whereas a 3-factor model requires a more disciplined assessment.
Consider whether or not to employ a FMEA table. Another interesting aspect of CLSI EP23A methodology is the omission of the standard application tool which is called Failure Modes and Effects Analysis (FMEA). A FMEA table provides a list the possible failure modes and the rankings of each risk factor, as well as the overall assessment of risk by combining those rankings into a Risk Priority Number (RPN). In the context of Kahneman’s modalities, the use of a FMEA tool provides a framework for a logical and quantitative assessment and the discipline to avoid the shortcomings of an intuitive assessment.
Consider the choice of “probability of occurrence of harm” in the 2-factor model. The typical risk factors are probability of occurrence of a failure (occurrence), detectability of that failure (detection), and severity of harm from such a failure (severity). The factor “probability of occurrence of harm” in the CLSI 2-factor model is a simplification and in Kahneman’s terminology a “substitution” of one question for another. This is a “subtle” change that is easily overlooked, even though EP23A (section 7.2.1) acknowledges that the probability of occurrence of harm is actually a more complicated factor than the probability of occurrence of a failure:
“…The overall probability of harm from the use of many laboratory measuring systems may be substantially lower than the probability of the failure because not every instance of failure will lead to patient harm.”
One reason for expecting a lower probability is that the laboratory’s QC procedures are expected to detect such failures and prevent the reporting of erroneous test results. Such detection depends on the particular control that has been implemented and is formally taken into account by the detection factor in the 3-factor model, rather than be handled “intuitively” in the 2-factor model. The original questions about the probability of occurrence of failures and the detection of failures have been mentally simplified by substitution of a single factor. However, that factor – probability of occurrence of harm – is actually more complicated and difficult to estimate, even though it enables a “thinking fast” assessment.
Consider CMS’s initial guidance that risk assessment requires NO formal tools. CMS endorses the CLSI EP23A guidance, then advises that there is no requirement that laboratories follow the EP23 methodology or apply any formal risk assessment tools. The CMS memo of August 16, 2013, includes the following answer in the FAQ section about IQCP:
Will laboratories be required to use a process map, fishbone diagrams, formal risk assessment charts and protocols, etc. in their IQCP? No, CLIA will not require the use of these tools in the development of an IQCP.
This clearly allows laboratories to utilize the “thinking fast” approach for making intuitive judgments and decisions, rather than a disciplined approach utilizing a logical process and statistical thinking. In fact, this makes it clear that there actually is no detailed methodology to help laboratories develop scientifically valid QCPs. The lack of knowledge and experience with risk assessment tools and techniques is not even recognized by CMS/CLIA as a major limitation in their new approach to QC.
What’s the point?
Kahneman identifies some conditions that lead people to employ intuitive judgments rather than more reasoned “statistical” thinking:
- When they are engaged in another effortful task at the same time;
- When they are in a good mood because they just thought of a happy episode in their life;
- If they score low on a depression scale;
- If they are knowledgeable novices on the topic of the task, in contrast to true experts;
- When they score high on a scale of faith in intuition;
- If they are (or are made to feel) powerful.
Busy laboratory analysts, already overloaded by “doing more with less,” certainly qualify as being engaged in another effortful task, all the time! They maintain an upbeat attitude and continually demonstrate professional responsibility and competence, but they are likely to be susceptible to intuitive judgments because of the pressure of the workplace.
Organizations like CLSI and CMS suffer from some of the other conditions. They are knowledgeable novices in the area of risk assessment, yet voice great faith in intuitive judgments and decisions. They tell us that risk assessment is nothing new, that it is something we already do in our everyday life and in our laboratories. If that’s the case, then it certainly reflects our intuitive mode of thinking, rather than the disciplined approach that is needed for risk assessment. They suggest that any QC will do as long as it’s right for you! But, what about our patients? What QC is right for verifying the attainment of the quality required for the intended use of patient test results?
We caution you to consider Kahneman’s concluding advice:
The way to block errors that originate in System 1 is simple in principle: recognize the signs that you are in a cognitive minefield, slow down, and ask for reinforcement from System 2… Unfortunately, this sensible procedure is least likely to be applied when it is needed most… The voice of reason may be much fainter than the loud and clear voice of an erroneous intuition, and questioning your intuitions is unpleasant when you face the stress of a big decision.
For laboratories, Risk-Based QCPs are a cognitive minefield and the best advice is to slow down and not rush into applications until you have a sound, disciplined methodology for risk assessment.
References
- Kahneman D. Thinking, Fast and Slow. Farrar, Straus and Giroux, NY, 2011.
- CLSI C23A. Laboratory Quality Control Based on Risk Management. Clinical and Laboratory Standards Institute, Wayne, PA, 2011.
- CMS Memo 13-54-CLIA. Individualized Quality Control Plan (IQCP): A New Quality Control Option. August 16, 2013.