Quality Requirements and Standards

I was privileged to participate in the Symposium “Top of Diabetes Diagnostics,” that was held in Zwolle, Holland, on September 2001. This special conference tackled not only harmonization, standardization, biologic variability, and tight glycemic control, it also covered the quality that HbA1c methods are currently achieving. Considering that diabetes is one of the major world health problems, the findings of this conference are particularly important

“Top of Diabetes Diagnostics” Symposium

James O. Westgard, PhD

• Symposium
• Problems with NGSP Certification
• Problems with waived testing
• Evidence of current performance
• References

Groningen University.

I was privileged to participate in the Symposium “Top of Diabetes Diagnostics,” that was held in Zwolle, Holland, on September 20, 2001.   This symposium was organized by Dr. Dikkeschei and Dr. Slingerland from the Isala Klinieken in Holland and was followed the next day by Dr. Erna Lenters doctorate ceremony at Groningen University, where I was also privileged to participate.  The celebration of Dr. Lenter’s research accomplishments was a memorable event, as shown by the pictures included here.

Procession

Symposium

The Symposium was chaired by Dr. Cas Weykamp, Chairman of the IFCC HbA1c Network, and included the following presentations:

• “Global harmonization” by Professor Gary John from Norfolk and Norwich University Hospital, UK;
• “Hemoglobin A1c standardization, analytical performance and Interpretation” by Dr. Erna Lenters,
• “Biological variability of minor human hemoglobins” by Prof. Andrea Mosca of the University of Milan;
• “The holy grail in glucose measurement” by Dr. Robbert Slingerland;
• “Tight glycemic control and quality of glucose meters” by Prof. David Bruns from the University of Virginia;
• “Alignment of quality goals, performance characteristics, and operating specifications”, an update on discussions of HbA1c goals that you can find on this website.

On the very left: Dr. David Bruns and Dr. James O. Westgard

The presentations documented the progress that has been made in harmonization of HbA1c measurements, but also emphasized the need for ongoing improvements, both for HbA1c and glucose monitoring.  In the context of the US marketplace, we have some particular problems with NGSP certification and “waived” testing that require some serious consideration to assure HbA1c test results are clinically useful, consistent over time, and comparable across methods and laboratories.

Problems with NGSP Certification

Readers of this website should be aware of the important work by Dr. Lenters and Dr. Slingerland in demonstrating the performance of current HbA1c Point-of-Care devices, which satisfy NGSP criteria for certification as “equivalent”, yet show biases from lot-to-lot that are large enough to invalidate their use for the intended clinical applications for diagnosis and monitoring [1].  The NGSP protocol calls for testing a single reagent lot once a year, then relies on the manufacturer to assure that other reagent lots are consistent and provide uniform  performance.  Unfortunately, Lenters and Slingerland’s data show that performance varies significantly from one lot to another.

Dr. Erna Lenters-Westra

Another limitation with the NGSP protocol is that manufacturers can choose the comparative method that is expected to show the best agreement.  Three comparative methods are available that align with the general measurement principles in use today.  Selection of the same measurement principle will naturally provide the best comparison results.  To provide more definitive comparison studies,” Lenters and Slingerland utilize the average of all three reference methods to provide a better estimate of the true values of samples. 

In addition, the NGSP criterion for acceptable performance is not as demanding as the clinical criterion for patient monitoring or the CAP criterion for Proficiency Testing.  In clinical interpretation of HbA1c results, a difference of 0.5 %Hb in two serial test results is considered to be clinically significant and is supposed to trigger a re-evaluation of treatment.  Given a treatment goal of 7.0 %Hb, a change of 0.5 %Hb  corresponds to 7.1%.  By comparison, the NGSP allowable difference is 0.75 %Hb, which is 10.7%.  NGSP acknowledges that the certification criterion must be tightened, but apparently it will take some time to accomplish this [2]. 

To address these problems, NGSP needs to improve its certification process, experimental protocols, and data analysis practices.  Such improvements should tighten the criterion for acceptable performance to 7.0%, test at least 2 reagent lots per year, utilize the average of reference methods for comparison, and re-evaluate the experimental conditions (number of specimens, data calculations) to assure that “certification” also indicates clinically acceptable performance.

Problems with Waived Testing

Many of these POC devices are “waived” by the FDA and therefore can be introduced into service with little validation and little ongoing monitoring of the quality of performance.  Users are supposed to follow the manufacturer’s directions for QC, but remember that FDA does not check the adequacy of the manufacturer’s QC directions so there is no assurance that the recommended QC is adequate and effective.  Thus, such waived POC devices likely operate with little or no QC. 

Another critical weakness in the US CLIA regulations is that waived tests are not required to undergo proficiency testing, thus there is little data available to document their actual performance in the field.  We assume their performance is acceptable if they have been certified by NGSP, yet Lenters and Slingerland have clearly shown that is not the case.  Performance of many POC devices changes from one reagent lot to another and are not likely to consistently achieve the certification requirements.  Thus, NGSP certification does not assure that on-going performance remains acceptable and there is no PT monitoring to help identify reagent lots and devices with poor performance.  Users must assume responsibility for careful monitoring through quality control, but that requires competencies that may NOT be present in many POC environments.

Some will argue that POC measurements will not be used for diagnostic decisions, only for patient monitoring, therefore additional QC and PT procedures are not necessary.   The problem is that the monitoring criterion – a change of 0.5 %Hb or 7.1% at the treatment goal of 7.0 %Hb – is actually more demanding than the diagnostic criterion!   [3, see previous update on HbA1c quality goals and performance requirements.]  Calculations based on the Reference Change Value (RCV) demonstrate that method CVs of 2.0% or less are needed (assuming within subject biologic variation of 1.0%).  While bias should not be a factor if the same method is used in serial measurements, any changes in methods across laboratories or over time will further impact the ability of monitor patients.

Evidence of Current Performance

Finally, here’s the latest 2011 CAP PT data on the performance of HbA1c methods.  Each point represents a method subgroup whose bias and CV are displayed on a Method Decision Chart.  The chart was prepared for a TEa of 7.0%, as shown by the y-intercept. The 3 lines describe performance levels of 4-sigma, 3-sigma, and 2-sigma, as can be seen from the x-intercepts (7/4 or 1.75, 7/3 or 2.33, 7/2 or 3.5). 





Of the 3026 laboratories participating in this CAP survey, 7% perform at 3-sigma or better, 27% between 2-sigma and 3-sigma, and 66% at less than 2-sigma.  It is apparent that the tightening of the CAP criteria from 15% to 12% to 10% to 8% and finally to 7% has occurred faster than manufacturer’s capabilities of improving the performance of their methods.  Nonetheless, the quality requirement of 7% represents the current intended clinical use, and clearly, many methods do not provide the necessary quality.

Knowing the state of diabetes testing doesn't mean we despair. It means we resolve ourselves to work harder than ever to improve it.

Thanks again to Dr. Erna Lenters-Westra for her invitation to this important symposium and for her work in the pursuit of better quality.

Dr. Erna Lenters-Westra with her diploma and symposium participants.

References

1. Lenters-Westra E, Slingerland RJ.  Six out of eight hemoglobin A1c point-of-care instruments do not meet generally accepted analytical performance criteria.  Clin Chem 2010;56:44-52.
2. Little RR, Rolfing CL, Sacks DB.  Status of hemoglobin A1c measurements and goals for improvement:  From chaos to order for improving diabetes care. Clin Chem 2011;57:205-214.
3. Westgard JO. Update on HbA1c quality goals and performance requirements. www.westgard.com/quality-hba1c-2011.htm