Recommendations for Validation of User QC |
|
Recommendations for Validation of User QC |
|
September 2003
An updated version of this essay appears in the Nothing but the Truth about Quality manual
James O. Westgard, PhD, FACB
I recently purchased a copy of ISO/DIS 15198: Clinical laboratory medicine – In vitro diagnostic medical devices – Validation of manufacturer’s recommendations for user quality control [1]. It’s available from NCCLS for $55, but don’t waste your money - it doesn’t say anything! The document is only 4 pages long and one page is definitions and another page references. I’ll summarize the other useless information in the discussion here.
[Note: Don’t confuse 15198 with 15189, though the numbers are similar. ISO/DIS 15189 has to do with “particular requirements for quality and competence” in medical laboratories, which is aimed at service laboratories, not manufacturers. This one – 15198 – is specifically intended to help manufacturers help laboratories do the appropriate QC for their methods.]
The document acknowledges that “the manufacturer of an IVD device is expected to provide specific, validated recommendations for quality control” and states that “this standard for manufacturers of IVD devices describes how to develop and validate recommendations for quality control to users.” To fulfill this obligation, here’s what a manufacturer should do (page 3 of said document):
- “Validation planning. The manufacturer of an IVD device shall develop a validation plan for quality control recommendations that require validation.”
- “Validation responsibility. The manufacturer shall assign the responsibilities for the validation taking into consideration that different disciplines may be necessary for different types of products or for different types of validation needs or phases (planning, protocols, execution, review, etc). Those responsible for it shall review the validation for comprehensiveness, statistical appropriateness and clinical appropriateness. A cross-functional team of experts may be formed to oversee the validation activities.”
- “Elements of the plan. Validation plans for quality control schemes shall consider the outputs from the Risk Analysis tools the Manufacturer may have employed during the design and development of the Device. Validation studies shall be designed to demonstrate the effectiveness of the QC procedures recommended by the manufacturer. Acceptance criteria linked to the performance claims shall be established prior to data collection. All statistical tests and sample size calculations shall be justified. These elements shall be documented in a validation protocol, which shall be approved prior to data collection.”
- “Performance evaluation studies. Results from internal and external evaluation studies shall be considered and, where appropriate, incorporated into the control scheme. Where a manufacturer enlists external sites to develop or validate recommendations for user quality control, such sites should, to the extent possible, reflect the user environments associated with the intended use of the device.”
- “Validation summary. The last stage is the preparation of the Validation Summary, which provides a synopsis of the validation objectives and data. The validation summary shall show that all of the acceptance criteria have been met. If the decision is made to make adjustments and or changes to the validation design, the changes shall be validated.”
- “Manufacturer’s records. The manufacturer will maintain records in accordance with ISO 9001, Quality management systems – Requirements and/or ISO 13485 Quality Systems, Medical Devices – Particular Requirements for the Application of ISO 9001.”
- Product information. The instructions for use (labeling) shall specify the validated quality control recommendations.
In short, the manufacturer is responsible for developing a plan, following that plan, preparing a report, maintaining records, and providing QC instructions to the user through proper labeling of the product. These guidelines are generic enough to apply to anything, but lack the details for developing useful and objective QC recommendations for laboratory methods.
I’m sure there were good intentions behind this document, but the document itself does not provide enough guidance to help manufacturers do what’s right and useful. I suppose we should be thankful that this document doesn’t inhibit or prevent doing what’s right and useful, but that seems like such a small accomplishment when the opportunity was so great.
The need for guidance on QC recommendations has been an issue for over ten years now. The issue came to the surface when the original CLIA rules were published in 1992 [2]. This “final” CLIA rule contained a provision for the FDA to validate the QC instructions (claims) of manufacturers. FDA first called a time-out to consider what to do, solicited information from users and manufacturers, developed a draft “guidance to manufacturers” document, which was aggressively opposed by manufacturers through the Health Industry Manufacturers Association (HIMA). After some time, FDA decided they just wouldn’t do it, in effect, punting the ball. ISO caught the ball, recognized a good opportunity to make a gain, but now has fumbled.
It’s interesting that earlier drafts of the ISO document provided more specific guidance that actually pointed manufacturers in the right direction. For example, I referenced recommendations from an earlier draft in a paper on “Validation of iQM active process control technology” [3], citing the information that manufacturers should include in recommendations to users, such as:
(a) the type of error that the quality control procedure is intended to detect;
(b) control materials that may be used,
(c) control materials that may not be used,
(d) recommended analyte concentrations,
(e) guidelines for determine acceptability criteria (control limits), and
(f) the probabilities of detecting and not detecting an inaccurate result.
Those guidelines were actually very helpful in providing direction on how to validate a new QC technology. Unfortunately, the final guidelines are a big step backwards. That probably reflects the compromises that had to be made to achieve consensus and a successful vote on the final document. The ultimate consensus outcome - agreement on nothing that is useful.
Just because ISO says so, it doesn’t mean the recommendations are either useful or correct. In this case, I would characterize the ISO guidelines as Inadequate but Still Official, as well as Irritating but Still Official. The “still official” is the big problem. Right or wrong, there are consequences for us, our laboratories, and the field of laboratory medicine.
The 15198 document is not the first indication that the ISO process is not working well for guidelines in laboratory medicine. A related document from the same ISO TC 212 group – recommendations for setting analytical performance goals [ISO/TR 15196, 4] - showed significant problems several years ago and led to the organization of the 1999 “Stockholm Consensus Conference on Quality Specifications in Laboratory Medicine” [5] by representatives of IUPAC, IFCC, and WHO to establish a global consensus position independent of ISO. It required adamant and aggressive public attack from the scientific experts in the field to influence ISO’s approach and recommendations.
Another example is the ISO position on “trueness and uncertainty” which has been discussed earlier on this website [To be uncertain or in error: That is the question] and also in the scientific literature [6]. That battle continues with no sign that ISO listens to the opinions and recommendations from scientific leaders in the laboratory medicine field. Even the European opposition is increasing and becoming more vocal.
It’s not hard to pinpoint the problem. While the ISO standards for laboratory medicine are administered by NCCLS under the guidance of ANSI (American National Standards Institute), the ISO process is not as open as the NCCLS process. While the objectives are similar – to achieve consensus on a position statement, the NCCLS committees have a better balance of the interested and affected parties. Outside scientific experts don’t seem to have as much influence in developing ISO guidelines as they do in NCCLS guidelines. “Outside” means outside of business, industry, and the government.
Take a look at the membership of the ISO working groups and see what you think about the balance of power. You’ll find that it’s hard to even find the committee membership lists (try www.iso.cn, www.nccls.org, or www.ansi.org). The printed NCCLS directory from 1999 shows that approximately 60 out of 80 voting members of US TAG 212 are from industry (the Technical Advisory Group in charge of 15189 and other documents related to laboratory medicine). That’s 75%! The other 25% are divided between representatives from government (CMS, CDC, FDA, NIST, a couple state depts. of health) and professional organizations (AACC, NACB, AABB, COLA, CAP). Contrast that with the NCCLS practice of generally having 1/3 from industry, 1/3 from government, and 1/3 from academic and user organizations.
Given that FDA has punted the ball and ISO has now fumbled, the best thing for a laboratory to do is get back into the game and take responsibility for the quality of its own testing processes! That means the laboratory itself must evaluate method performance, assess the potential sources of errors, and monitor and control those errors. You can’t depend ISO to help with this! You can’t depend on government regulations to take care of this! You can’t depend on manufacturer’s to do this for you! You have to do this yourself to assure the quality of testing for your customers, physicians, and patients.
How to do it - there’s a lot of practical guidance available right here on this website. Start with the following lessons and discussions:
- CLIA Final Rule: QC - Quality or Compliance? http://www.westgard.com/cliafinalrule3.htm
- Six Sigma quality management and desirable laboratory precision. http://www.westgard.com/essay35.htm
- Six Sigma quality management and requisite laboratory QC. http://www.westgard.com/essay36.htm
- Formulating a Total Quality Control strategy. http://www.westgard.com/lesson55.htm
These lessons and discussions provide a framework for you to build an effective quality system to monitor and control the analytical testing processes in your laboratory. While it would have been nice to have someone else take care of this for us, that’s still a dream! The ISO 15198 document reveals the reality of the situation. We are still responsible for the quality of testing in our laboratories. Our knowledge and skills are critical for assuring the quality of test results for our patients. We’re still needed to do the right things right!
- ISO/DIS 15198 Clinical laboratory medicine – In vitro diagnostic medical devices – Validation of manufacturer’s recommendations for user quality control. Available from NCCLS, 940 West Valley Road, Suite 1400, Wayne, PA.
- US Dept. of Health and Human Services. Medicare, Medicaid, and CLIA Programs: regulations implementing the Clinical Laboratory Improvement Amendments of 1988 (CLIA). Final rule. Fed Regist 1992;57:7002-186.
- Westgard JO, Fallon KD, Mansouri S. Validation of iQM active process control technology. Point of Care2003;2:1-7.
- ISO/DIS 15196. Determination of analytical performance goals for laboratory procedures based on medical requirements. Available from NCCLS, 940 West Valley Road, Suite 1400, Wayne, PA.
- Strategies to set global analytical quality specifications in laboratory medicine. Special Issue. Scan J Clin Lab Invest 1999;59(no. 7).
- Hyltoft Petersen P, Stockl D, Westgard JO, Sandberg S, Linnet K, Thienpont L. Models for combining random and systematic errors. Assumptions and consequences for different models. Clin Chem Lab Med 2001;39:589-595, 2001.
Other Essays:
