Talking and tackling Risk-Based QC Plans

The clock is ticking down for EQC. CMS stated in an initial memo on EP23 that Risk-based QC would replace EQC protocols at the end of 2013, and that by 2014, EQC would no longer be an acceptable laboratory procedure. While the CLSI EP23 protocol has been out for more than a year, though, CMS has yet to issue any regulations or guidelines on how to implement, comply, or inspect Risk QC. What's going on?

Talking and Tackling Risk-Based QC Plans

James O. Westgard, PhD
April 2013

I attended the CLSI workshop on “Tools for Tackling EP23” [1] last week in Chicago (April 19, 2013). My interest was to keep up-to-date on the CMS timetable for implementing Individualized QC Plans and CLSI’s training to support such implementation. The good news – this was the 5th workshop that CLSI has presented and they are ramping up their education and training activities. The bad news – CMS has still not established a timetable for implementing Individualized QC Plans (IQCP) nor published “interpretive guidelines” for implementation. That leaves laboratories and accrediting organizations wondering when and what to do!

Everyday risk management

In the workshop, CLSI emphasized that Risk Assessment is nothing new. We do Risk Assessment all the time, at home and at work. We assess and evaluate risks on a daily basis so it should be easy to apply Risk Assessment to QC in the laboratory.

It happened that the workshop was the day after Chicago received 8 inches of rain, which caused extensive flooding throughout the area. Over 500 flights were cancelled at O’Hare airport and highway travel was slow and interrupted by detours. Homes and cars were damaged due to the high waters. This is a good example of risk in our daily lives that resulted in harm in the form of inconvenience for travelers and property damage for residents.

So what failed in Chicago's Risk Assessment? Had they improperly assessed the risk of 8 inches of rain at one time? Had they not implemented appropriate controls to mitigate the risk of floods? Given the many people who had water in their basements, cars that were damaged by being submerged, and the lack of insurance to cover these damages, I’m not so sure that Risk Assessment is so simple. And it's pretty clear that people don't do a great job of assessing risk at home and at work. There's also the issue of whether an individual has the capability of managing some risks, or whether the government and industry should bear the major responsibility for some risks.

Risk management of medical devices

Manufacturers are required to assess the risks of new medical devices and to provide that information to the FDA as part of the 401(k) approval process. In industrial applications, risk is a function of occurrence (OCC), detection (DET), and severity (SEV). These 3 factors are often assessed on a scale from 1 to 10, then risk is estimated by calculating a Risk Priority Number, RPN, which is the product of OCC*SEV*DET. Typically, a FMEA table (Failure Modes and Effects Analysis) is provided to identify potential failure modes, the effect of such failure, a rating of severity, expected probability or frequency of occurrence, procedure to control for that failure, detection available by the control procedure, and the estimated risk. These are very detailed and thorough assessments of risk that are required according to ISO 14971 [2].

Unfortunately, the details of the manufacturers risk evaluation are not available to laboratories, except for information that is included in product labeling. CLSI had originally proposed another document (EP22) to provide guidance to manufacturers on how to present risk information to laboratories. However, that document committee is no longer in existence. That leaves laboratories in a difficult situation. They need to evaluate risk, but they lack key information on the performance of the manufacturer’s control mechanisms. It was mentioned at the workshop that CLSI is developing an electronic worksheet that could be used by manufacturers to inform laboratories of their risk studies, but it was also clear from comments by the speakers that there is little enthusiasm from manufacturers to provide such information.

Simplified model for laboratory applications

Meanwhile, back at the laboratory, CLSI is recommending a simplified approach to estimate risk from the probability of occurrence of harm and the severity of harm. Note that this is a 2-factor risk model that omits the factor of detection. As we’ve discussed previously on this website, that is a curious simplification. If the purpose of Risk Assessment is to develop a QC Plan to detect errors, why eliminate or ignore the detection factor in Risk Assessment?

There is another simplification that is more subtle. The guidance equates the probability of occurrence of harm with the frequency of occurrence of failures and recommends a simple semi-quantitative scoring scale based on ISO 14971 [2]:

• Frequent – once per week (5)
• Probable – once per month (4)
• Occasional – once per year (3)
• Remote – once every few years (2)
• Improbable – once in the life of the measuring system (1)

On this scale, one might estimate the frequency of occurrence on an 8 inch rainfall in Chicago as “remote” or even “improbable”, thus the risk of flooding is very low and the probability of harm would likewise be considered low. But here’s the problem! Estimating the frequency of an event is not a good measure of the impact of the event. For example, the harm from flooding depends on the number of people affected by the event. Eight inches of rain in the middle of nowhere does not create the same risk as eight inches of rain in Chicago. Population density is an important factor in determining harm. Likewise, in laboratory testing, an infrequent event that affects a high throughput analyzer creates a higher risk of harm than the same frequency event occurring with a unit-use testing device, where only one patient may be harmed. Frequency of events does not equate to the number of defective test results that may be produced. This is a serious limitation of the simplified risk assessment methodology.

Intended use?

One can understand how this happened. Remember that the risk-based QC Plan was developed to overcome the limitations of Equivalent Quality Control (EQC). EQC was motivated by the desire for simpler QC procedures for Point-of-Care testing, particularly for unit-use devices. Thus, the context for development of Risk-based QC is Point-of-Care testing. However, CMS is recommending Individualized QC Plans as an alternative approach for meeting the CLIA QC requirements for any and all methods [3].

Laboratories can choose to deploy IQCPs or implement default QC, which was described as the CLIA minimum of 2 levels of controls per day and referenced to 42 CFR 493.1256(d)3 in the CLIA regulations [4]. Given that the original purpose of EQC was to eliminate “one size fits all” QC, it is interesting that it remains the default option and will likely continue to be the standard practice in many laboratories.

The difficulties in implementing IQCPs are many, starting with the education and training that is needed to adopt and adapt new concepts and practices, the lack of information from the manufacturer to help laboratories understand critical failure-modes and the detection capabilities built into analyzers, the subjective nature of the risk assessment with its qualitative scales and arbitrary judgments on risk acceptability, the lack of consideration of the detection capabilities of different control mechanisms, and finally the limited ability of accrediting organizations to assess whether laboratory IQCPs really do provide “equivalent quality testing.” Instead, accreditors will likely have to check whether a laboratory follows its IQCP, without any assessment of whether the plan contains the right controls and whether those controls really work. Inspectors may not even have the power to challenge the laboratory on its Risk Assessment. They may have to accept that any QC will do as long as it is written down in the laboratory’s IQCP.

If this comes to pass, IQCPs will certainly provide laboratories and testing sites with a lot of "flexibility" for implementing various control mechanisms! Effectiveness of these controls will be a different issue. CMS relies on physician complaints as a measure of the quality of laboratory testing. If they hear no complaints, they assume there is no problem with quality. Unfortunately, no complaints may simply mean there is no evidence of anything at all! CMS further assumes that inspectors are able to “see” problems with quality without access to any quantitative data on which to make an objective assessment. If daily SQC results are not available to monitor performance, then periodic monitoring via proficiency testing should be extended to all testing sites and all categories of tests, including waived tests. If daily QC gets reduced or eliminated, we need at least some other objective data on the quality of testing that is actually being achieved.

Future of “QC for the Future”

Eight years after the conference on “QC for the Future“ [5], the future of QC may be moving away from statistical QC procedures that can be designed to detect medically important errors. Instead, the future of QC may be moving toward qualitative and subjective Risk QC Plans that require no validation of their monitoring capabilities. “Any QC will do” may become the standard practice in US laboratories, replacing “one size doesn’t fit all QC”. Or, more likely, the two options will exist side by side for many years to come. “The Right QC” will remain elusive until “equivalent quality testing” is defined in quantitative terms and implemented by an objective and scientifically sound methodology.

References

1. CLSI EP23A. Laboratory Quality Control Based on Risk Management; Approved Guideline. Clinical and Laboratory Standards Institute, 940 West Valley Road, Suite 1400, Wayne PA, 2011.
2. ISO 14971. Medical devices – Application of risk management to medical devices. International Standards Organization, Geneva, 2007.
3. CMS memorandum on implementing the Individualized Quality Control Plan (IQCP) for Clinical Laboratory Improvement Amendments (CLIA). March 9, 2012. www.cms.gov/Medicare/Provider-Enrollment-and-Certification/SurveyCertificationGenInfo/Downloads/SCletter12-20-.pdf

4. US Centers for Medicare & Medicaid Services (CMS). Medicare, Medicaid, and CLIA Programs. Laboratory requirements relating to quality systems and certain personnel qualifications. Final Rule. Fed Regist Jan 24 2003;16:3640-3714.

5. QC for the Future. LabMedicine 2005;36(No. 10):609-640.