Sigma Metric Analysis
Evaluation of a blood glucose device
Glucose meters are coming under renewed scrutiny - off-label use with critcally ill patients, infection issues, and new draft guidelines for tighter requirements for FDA clearance. We take a look at a current glucose meter to see if current performance is meeting today's goals - and possibly tomorrow's...
Evaluation of a blood glucose self-monitoring system
- The Precision and Comparison data
- Determine quality requirements at the measured decision levels
- Calculate Sigma metrics
- Summary of Performance by Sigma-metrics chart
- A Concerning Coda
- Conclusion
MARCH 2014
Sten Westgard, MS
[Note: This QC application is an extension of the lesson From Method Validation to Six Sigma: Translating Method Performance Claims into Sigma Metrics. This article assumes that you have read that lesson first, and that you are also familiar with the concepts of QC Design, Method Validation, and Six Sigma. If you aren't, follow the link provided.] |
This Sigma analysis looks at a number of papers from 2011 to 2013 which examined the performance of a point-of-care (POC) glucose device:
SKUP analysis report OneTouch Verio, http://skup.nu/GetFile.ashx?fileid=474 SKUP/2011/86
A Comprehensive Evaluation of the Performance of the Test Strip Technology for OneTouch Verio Glucose Meter Systems, Bailey T, Chang A, Rosenblit PD et al, Diabetes Technology and Therapeutics 2012 Volume 14:8;701-709
The Imprecision and Bias Data
Below is the table of imprecision and bias for three levels:
OneTouch Verio |
Level |
CV% | Bias % |
Glucose | <7 mmol/L (<126.1 mg/dL) | 2.3% | 4.5% |
7 - 10 mmol/L (126.1 - 180.2 mg/dL | 3.1% | 2.9% | |
>10 mmol/L (> 180.2 mg/dL) | 3.5% | 0.1% |
Looking at the raw numbers, you may find it difficult to judge the method performance. From experience, you might be able to tell when a particular CV is high or low. But the numbers themselves don't tell the story.
If we want an objective assessment, we need to set analytical goals - specify quality requirements - and use those to calculate the Sigma-metric.
Determine Quality Requirements
Now that we have our imprecision and bias data, we're almost ready to calculate our Sigma-metrics. But we're missing one key thing: the analytical quality requirements.
In the US, traditionally labs look first to CLIA for guidance. But for glucose, there are two standards that are acceptable. In the US, CLIA holds glucose at the core laboratory to a tighter standard than at the point-of-care. Indeed, for much of the glucose range, the core lab methods are expected to meet basically a 10% quality requirement, while point-of-care blood glucose meters need only meet a 20% quality requirement. Given that almost all other soruces of quality requirements have more stringent goals, the minimum achievement of these POC devices should be able to meet the CLIA specifications for total allowable error (TEa).
However, the FDA issued some new draft guidelines for glucose meters in early 2014, and it's of interest to determine whether or not this meter would pass these possible future guidelines. Since the device is already on the market, it won't be necessary to meet this standards, but as a laboratory user, it is good to know if the device you are using will meet future standards or if you will want to replace it with something that meets those standards in the future.
Method | CLIA Goal |
Glucose |
Target value ± 6 mg/dL 20% (for POC devices) |
FDA Draft Guidance for glucose meter clearance requirements:
- 99% of all values fall within ± 10% of the reference method for glucose concentrations > 70 mg/dL (3.88 mmol/L) and within ± 7 mg/dL for glucose concentrations < 70 mg/dL
- No individual result should exceed ± 20% of the reference method > 70 mg/dL and outside ± 15 mg/dL for glucose concentrations < 70 mg/dL.
Calculate Sigma metrics
Now the pieces are in place. Remember the equation for Sigma metric is (TEa - bias) / CV.
Example calculation: for glucose at the lower level range below 7.0 mmol/L, the current POC goal is 20%. We also know from the study that at that level, the Verio has 2.3% imprecision and a 4.5% bias:
(20 - 4.5) / 2.3 = 15.5 / 2.3 = 6.7 Sigma
If we were to apply the CLIA 10% goal:
(10 - 4.5) / 2.3 = 5.5 / 2.3 = 2.39 Sigma
If we were to apply the FDA draft guidance:
We would need the meter to hit 3.9 Sigma at the CLIA goal of 10% and above 6 Sigma for the POC goal of 20% at all levels
ll that in industries outside healthcare, 3.0 Sigma is the minimum performance for routine use. 6.0 Sigma and higher is considered world class performance.We'll simplify the table below and calculate all the Sigma-metrics.
OneTouch Verio |
Level |
CV% | Bias % |
POC Sigma |
CLIA Core Lab Sigma |
2014 FDA Draft |
Glucose | <7 mmol/L (<126.1 mg/dL) | 2.3% | 4.5% | 6.7 | 2.49 | Fail |
7 - 10 mmol/L (126.1 - 180.2 mg/dL | 3.1% | 2.9% | 5.5 | 2.3 | Fail | |
>10 mmol/L (> 180.2 mg/dL) | 3.5% | 0.1% | 5.7 | 2.8 | Fail |
While under current POC standards, this meter is more than acceptable, it looks like it would not be cleared if the new FDA draft guidelines were imposed.
In a separate study, It was found that for healthcare professionals, below 75 mg/dL (or 4.2 mmol/L), only 93.5% of results fell within 10 mg/dL of the reference method result, but above 75 mg/dL, 100% of results fell within 20% of the reference method. This matches the table of data that we see here - the method is better at achieving success at a 20% target than it is for 10% target..
Summary of Performance by Sigma-metrics Method Decision Chart
If the numbers are too much to digest, we can put this into a graphic format with a Six Sigma Method Decision Chart. We'll use a normalized OPSpecs chart here so we can show both the performance meeting the 10% goal and the 20% goal.
So, if we simply judge this using today's 20% POC glucose goal, this is an excellent to world class method. If we want to judge it against the same standards as a core laboratory method, it's not acceptable. If we want it to meet the possible future FDA quality goals, it probably won't pass.
If we had this method today, what would we do for QC? In this case, we plot the method data on an OPSpecs (Operating Specifications) chart. However, keep in mind that since this is a POC device, there might be little flexbility in the QC that can be done on the device.
If we're simply using today's POC goal of 20% allowable total error, we can use 3s limits and run 3 controls. If we're going to hold this method to "lab quality" standards, we're goingto need possibly double the number of controls and use all the "Westgard Rules" we can muster.
Conclusion
The studies conclude that this method is acceptable, and under current guidelines, it is. In fact, compared to many other glucose meters on the market right now, this has superior performance. But the possible future FDA requirements will be more stringent, and if this device was to be submitted under those future targets, it might not gain clearance.