Maryland General

July 2006
with Sten Westgard, MS

As a result of the GAO investigation, there are 13 recommendations being made in their report. We've analyzed each point to determine if they're useful, bizarre, or just irrelevant.

1. Standardize survey findings.
2. Ensure consistent advance notice by states
3. Ensure identifying deficiencies comes before education
4. Impose sanctions on labs with consecutive failures
5. Require all survey organizations to require whistle-blower posters
6. Require quarterly proficiency testing
7. Ensure timely evaluation of survey organizations
8. Ensure that new survey rules are reviewed in a timely manner
9. Hire more staff
10. Perform more validation inspections of state survey organizations
11. Require more independent validation of survey organizations
12. Collect and review more survey data
13. Establish an enforcement database

• Conclusion

The GAO report on CMS delivered 13 recommendations for improvement. We're going to go through each point, explain why GAO made this recommendation, include relevant quotes from the report, and provide a little commentary on each recommendation.

1. Work with exempt-state programs and accrediting organizations to standardize their categorization and reporting of survey findings in a way that tracks to CLIA inspection requirements and allows for meaningful comparisons across organizations, such as the analysis of trends in the citation of condition-level deficiencies.

Why is this being recommended?
"Determining the quality of lab testing is difficult because it is virtually impossible to crosswalk inspection requirements across survey organizations. Without standardized survey findings across all survey organizations, CMS cannot tell whether the quality of lab testing has improved or worsened over time or whether deficiencies are being appropriately identified" [page 45]

GAO was not able to merge the data on laboratory deficiencies, because sometimes what was considered a serious violation in one state or organization was not considered serious in another state or organization. Given the wide differences between just CAP and JCAHO, not to mention the differences in the state-level regulations, it would be useful to have a way to compare across all entities. And certainly it would be good to have a way to determine the number of serious deficiencies occurring nationwide.

The danger here is that if the Congress forces CMS to impose strict standards, the survey organizations may be forced to abandon the requirements that make them distinct from (and possibly better than) their competitors. Would CAP have to give up checklists? Would JCAHO have to abandon the Tracer methodology? Would both of them have to work off the exact same list of questions and topics? A very strict standardization might even eliminate any differences between the offerings of these groups. In fact, if a strict standardization is actually mandated, they might as well just bring this back in-house and make it a purely government function, instead of a partly private one.

2. Ensure that the advance notice of upcoming surveys provided to physician office labs is consistent with CMS's policy for advance notice provided by state agencies.

Why is this being recommended?
There's a big difference in the amount of advanced notice for inspections. Of course, CAP and JCAHO have shifted to unannounced inspections now, but the Washington-state program provides up to 4 weeks notice, and COLA provides up to 12 weeks advanced notice of inspections. [page 21] Officially, CMS guidance is that only two weeks notice is allowed for state survey agencies.

Since COLA inspects primarily physician office labs, which operate differently than hospital labs, there may be a good reason to give them advance notice: "(1) ensuring that the lab is open and that appropriate personnel are available to answer surveyor's questions, and (2) minimizing disruptions to patient care" [ibid.] Of course, these are also concerns for all laboratories.

Really, all that GAO is asking for is consistency. If CMS changes their guidance so that it matches COLA and state practices (or vice versa), and justifies why physician office labs deserve advance notice while hospital labs do not, then everything will be fine.

3. Ensure that regulation of labs is the primary goal of survey organizations and that education to improve lab quality does not preclude the identification and reporting of deficiencies that affect lab testing quality.

Why is this being recommended?
GAO seems to believe that sometimes CMS is "educating" instead of sanctioning labs with deficiencies:

"CMS appears to be inappropriately stressing education over regulation in its implementation of (1) 2003 lab quality control requirements for the CLIA program and (2) proficiency testing for lab technicians who interpret Pap smears, a test for cervical cancer....

"In part because of a lack of lab "buy in" for some of the new policies and procedures, CMS officials have extended the educational period for about another 2 years. CMS has taken a similar educational approach to Pap smear proficiency testing, which began in 2005. CMS will not cite deficiencies or impose sanctions against labs in which staff fail the new Pap smear proficiency testing in 2005 or 2006, as long as the labs and individuals involved complete such testing, including following the regulatory protocol for subsequent testing in the case of an initial failure." [page 24]

Here the GAO report is focused on the word "educational" when the real issue is the practical utility of the new regulations in question. Strangely, "EQC" isn't mentioned directly in the GAO report. CMS hasn't enforced those regulations because they aren't scientifically sound (that's the real "buy in" that labs are lacking). While CMS struggles to find regulations that are both sound and acceptable, they fall back on their "educational" role. But instead of educating laboratories, they're really just stalling for time, while they try to come up with a regulation that is both valid scientifically and acceptable to the laboratory community (and for EQC, getting both may well be impossible).

4. Impose appropriate sanctions on labs with consecutive condition-level deficiencies in the same requirements.

Why is this being recommended?
GAO believes that lab sanctions are rarely imposed - but they should be:

"Few labs were sanctioned by CMS from 1998 through 2004 - even those with the same condition-level deficiencies on consecutive surveys - because many proposed sanctions are never imposed.... [page 29]

"Although few labs were sanctioned from 1998 through 2004, over 9,000 labs had sanctions proposed during that same time period. Before sanctions go into effect, labs are given a grace period to correct condition-level deficiencies, unless the deficiencies involve immediate jeopardy, that is, an imminent threat to patient health and significant hazard to public health. Most labs correct the deficiencies within the grace period.... [page 30]

"From 1998 through 2004, 274 labs surveyed by state agencies had the same condition-level deficiency cited on consecutive surveys and 24 of these labs had the same condition-level deficiency cited on more than two surveys.... We found that only 30 of the 274 labs with repeat condition-level deficiencies had sanctions imposed - either principal, alternative, or both.... From 1998 through 2004, less than 1 percent of accredited labs (81) lost their accreditation; few of these labs were subsequently sanctioned by CMS and many still participate in the CLIA program." [page 31]

The list goes on. GAO found many examples where labs are getting second chances, third chances, and more, to get into compliance (Somewhat akin to the QC practice of "repeated, repeated, got lucky").

The deeper question is whether inspections are meant to be invasive and punitive or are they supposed to be for assurance purposes. We used to operate under the assumption that laboratories were staffed by professionals who were committed to quality - inspections were only to confirm that, to provide a means of assurance. The laboratory staff didn't need sanctions to motivate them. We now seem to be moving to a new paradigm where laboratories are no longer trusted. Inspections are meant to discover whether or not the staff is doing any quality at all. And the threat of swift, sure, and severe punishment is to be the new motivation for compliance.

If this is to be the new paradigm, then stronger enforcement will indeed be necessary. But "getting tough" on laboratories will not insure quality. You simply cannot inspect quality into a process, or scare people into doing a quality process.

5. Require all survey organizations to develop, and require labs to prominently display, posters instructing lab workers on how to file anonymous complaints.

Why is this being recommended?
The GAO believes "some lab workers may not be filing complaints about quality problems at their labs because of anonymity concerns or because they may not be familiar with filing procedures" [page 27]. They noted that when CAP required whistleblower posters put in all labs, in response to the Maryland General scandal, the number of complaints "more than doubled in 2004" and based on partial data of 2005, looked like it was about to double again. The progression went from 84 in 2003, to 170 in 2004, to 290 in the first 9 months of 2005. The proportion of those complaints that were substantiated fell each year, but there were over 74 complaints substantiated in the first 9 months of 2005.[page 290]

Posters are helpful, but they're at best treating a symptom of the problem. If lab workers are under duress, it's because quality has become a problem in one way or another. Posters will help get a few more whistleblowers, but that isn't going to stop the disease.

6. Consistent with CLIA, require quarterly proficiency testing, except when technical and scientific considerations suggest that less frequent testing is appropriate for particular examinations or procedures.

Why is this being recommended?
CMS went to proficiency testing three times a year quite a while ago. But evidently, they didn't do that the right way:

"CMS's decision to require proficiency testing for almost all laboratory tests only three times a year is inconsistent with the statutory requirement. CLIA requires that proficiency testing be conducted "on a quarterly basis, except where the Secretary determines for technical and scientific reasons that a particular examination or procedure may be tested less frequently (but not less often than twice per year)...." [page 33]

"The language of the statute, as well as relevant legislative history, indicate that a decision to reduce the frequency of testing should be in the nature of an exception made with regard to a particular test, not the norm for all tests, and must be based on 'technical and scientific' considerations related to that particular test. The reasons that CMS gave for requiring only three events per year were not based on scientific and technical considerations relevant to particular tests. Instead, CMS's decision was based on concerns of an administrative and logistical nature that CMS wanted to alleviate by reducing the frequency of testing events...." [page 34]

"CMS supplied a brief, undated narrative, which it attributed to the Centers for Disease Control and Prevention. It was not clear to us that this narrative was contemporaneous with the decision to reduce the frequency of proficiency testing. Moreover, the narrative focused on the relative costs and benefits of proficiency testing at various intervals. There was no analysis of technical and scientific considerations with regard to particular tests that presented a basis for reducing the frequency...." [page 52]

"Based on CMS's response, we maintain that CMS's decision to require proficiency testing three times a year is not authorized by CLIA."

Basically, CMS is not in compliance with CLIA! Even though they decided it was more practical to have laboratories perform proficiency testing three times a year, they didn't justify it legally in the required manner.

Will going back to quarterly proficiency testing solve the problem with laboratory testing today? Unlikely. It would raise costs dramatically. CMS has to get back into compliance with CLIA by providing the correct justification. Then we can get back to the real problems.

7. Ensure that evaluations of exempt-state and accrediting organization inspection requirements take place prior to expiration of the period to expiration of the period for which they are approved in order to ensure the continued equivalency of their requirements with CLIA's.

Why is this recommendation being made?
GAO found that CMS is behind a legally mandated schedule of approvals:

"We found that CMS has been late in determining that exempt states' and accrediting organizations' inspection requirements and processes are at least equivalent to CLIA's. CMS must verify their equivalency and, by regulation, CMS requires such survey organizations to seek reapproval at least once every 6 years, or more frequently if deemed necessary..."

"CMS has not completed its equivalency reviews within these time frames and accrediting organizations and exempt state programs have continued to operate without proper approval. Equivalency reviews for CAP, COLA, JCAHO, and Washington due to be completed between November 1, 1997 and April 30, 2001, were an average of about 40 months late." [page 35]

When confronted on this issue, CMS claimed that the lack of staff is causing those the delays. CMS has only 21 FTE staff, down from 29 at its peak. [Frankly, I didn't realize it was so small an agency.] To this, GAO points out that CMS has $70 million in lab fees that could be used to add staff. [page 36]

CMS needs to catch up on its To-Do list. Adding staff may help.

8. Ensure that changes to the inspection requirements of exempt states and accrediting organizations be reviewed prior to implementation, as required by regulation, to ensure that individual changes do not affect the overall CLIA equivalency of each organization.

Why is this recommendation being made?
GAO found that CMS may not be reviewing the changes of the accrediting agencies and exempt-state agencies, and the changes being made by those groups may not be in compliance with CLIA requirements:

"(1) accrediting organizations are required to submit changes to their inspection requirements and policies 30 days prior to changing their standards and (2) exempt-state programs are required to provide notice when they change their licensure or inspection requirements. Although federal regulations [ed. 42 CFR 493.573(a)(3)(2005)] require CMS to review equivalency when an accrediting organization or exempt-state program adopts new requirements, a CMS official told us that the agency is not required to review such changes before their implementation to ensure equivalency. As a result, such survey organizations may introduce changes that are inconsistent with CLIA requirements." [page 35]

Certainly CMS needs to make sure that all the different organizations implement rules and policies that are compliant to CLIA. But on some cases this hasn't been happening in a timely fashion (GAO example: JCAHO's 2004 inspection requirements went into effect six months before they were finally approved by CMS).

9. Allow the CLIA program to utilize revenues generated by the program to hire sufficient staff to fulfill its statutory responsibilities.

See Recommendation 7 above. GAO believes CMS needs more staff to get the job done.

10. Ensure that federal surveyors validate a sufficient number of inspections conducted by each state survey agency to allow a reasonable estimate of their performance, including a minimum of one independent validation review for each state survey agency surveyor.

Why is this recommendation being made?
GAO believes there isn't enough surveying of the surveyors to provide a reliable validation:

"CMS validation reviews... .are intended to evaluate lab surveys conducted by both states and accrediting organizations.... CMS requires its regional office to conduct validation reviews of 1 percent of labs inspected by state survey agencies in a year...." [page 37]

"The requirement to validate 1 percent of labs surveyed by state agencies in a year - roughly 100 validation reviews each year - does not ensure sufficient oversight of state survey agencies....While the 10 CMS regional offices generally validated 1 percent of the state survey agency inspections within their region, they often did not validate 1 percent of inspections within each state, and, in fact, performed none in some states." [page 38]

Will more inspections of inspectors help the quality of laboratory testing? It's an indirect attack on the problem at best. Certainly we need to make sure the inspectors are doing their job correctly. But insisting on a CMS validation for every state surveyor may not be the only solution to that issue.

11. Require that almost all validation reviews of each accrediting organizations' surveys be an independent assessment of performance.

Why is this recommendation being made?
Many CMS validation reviews are conducted simultaneously with the state surveyors, and the GAO believes this prevents a truly useful validation review:

"Seventy-five percent of validations of state lab surveys were conducted simultaneously from fiscal years 1999 through 2003. According to CMS officials, the large proportion of simultaneously validation reviews provides an opportunity for federal surveyors to share information with state surveyors, monitor their conformance with CLIA inspection requirements, and identify training and technical assistance needs. However, we found that such reviews do not provide an accurate assessment of state surveyors' ability to identify condition-level deficiencies."[pages 38-39]

The table provided by GAO in the report shows that simultaneous validations find very few problems missed by the state surveyors, but when the validation reviews are done independently, they find more deficiencies missed by the state surveyors.

An uncharitable comment would be to imply that the CMS officials are "educating" their state surveyors during the simultaneous inspections instead of finding fault with them. When the inspections are independent, however, they can't do any "educating"

Again, will more and better inspections of the inspectors help? A little. Are mandated independent validation reviews the only solution to the issue? No. Is this addressing the real problem with the quality of laboratory testing? Not really.

12. Collect and routinely review standardized survey findings and other available information for all survey organizations to help ensure that CLIA requirements are being enforced and to monitor the performance of each organization.

Why is this recommendation being made?
GAO believes CMS is not collecting enough data from the states and accrediting organizations and is not making use of the data it does collect:

"CMS does not routinely collect and analyze data essential for effective oversight of the CLIA program....

"it does not routinely track the proportion of labs, by state, in which state survey agencies identify condition-level deficiencies - those that denote serious or systemic problems.... As noted earlier in this report, variability in survey findings suggests inconsistencies in how surveys are conducted. CMS also does not require exempt-state programs and accrediting organizations to routinely submit data on serious deficiencies identified at the labs they inspect, unless the deficiencies pose immediate jeopardy to the public or an individual's health. As noted earlier, the lack of a common vocabulary on what constitutes a serious deficiency would make it virtually impossible for CMS to analyze such data...."

"We also found that CMS does not effectively use available data to assess clinical lab quality in areas such as proficiency testing, sanctions, and complaints.... [page 41-42]

CMS needs better data handling systems. And it's trying to address that issue already. It would be useful to have a centralized tool that can spit out summaries and breakdowns of deficiencies. That would give a more accurate picture of how bad the situation is in the laboratories of the country.

13. Establish an enforcement database to monitor actions taken by state survey agencies and regional offices no labs that lose their accreditation.

Why is this recommendation being made?
See recommendation 12 above. The main reason:

"CMS also lacks a complaints database, and therefore was unable to assess the impact of CAP's decision to require labs to prominently display a post on how to file a complaint." [page 42]

Conclusion: Irrelevance of GAO Recommendations for Improving Quality

Recall that the major issue was to find out if Maryland General was just “one bad apple” or if Maryland General is the tip of the iceberg, revealing a widespread systemic problem with the quality of laboratory testing throughout the US. If the answer is “one bad apple,” then the solution is to address the specific problems in that one hospital. If the answer is more widespread, then a broader strategy is needed to address the problem of quality in clinical laboratories.

The major issue could not be adequately assessed, however, because the lack of standardization of different survey programs made it impossible for GAO to determine the trends in citations of condition-level deficiencies. GAO instead relies on proficiency testing data to demonstrate that the quality of laboratory testing is not improving and may in fact be getting somewhat worse. Our own investigation of the PT data leads to estimates of 3 to 4 sigma quality (at best) for routine testing today, which indicates major improvements are still needed to achieve “world class quality.”

GAO then proceeds with the assumption that an improved inspection process is the solution to the laboratory problems with quality. If we accept that direction and assume that the inspection process can indeed be perfected, that only means that the detection of problems will be improved. GAO is assuming that enforcement and sanctions will then become a sufficient motivation for making the changes that actually solve the problems and improve the quality of laboratory testing.

To make the necessary improvements, laboratories will again be faced with “doing more with less” while continuing to meet ever-more demanding production schedules. And this occurs in the midst of a staffing crisis and a diminishing supply of well-trained analysts. There is no redress for CLIA-88’s lowering of personnel standards, the consequent turndown in MT and MLT training programs in the 90s, and the subsequent shortage of well-trained laboratory professionals today.

Everyone who works in a laboratory knows we have serious quality problems. We don’t need GAO to tell us that, but we could use some help and guidance to move in the right direction. Unfortunately, GAO doesn’t have that insight into quality management and can only audit the existing regulations and identify the shortcomings of the current laboratory inspection process. After almost two decades of laboratory regulation, it seems clear that CLIA inspections and compliance do not assure the quality of laboratory testing. Regulations must not be the right answer!

---

James O. Westgard, PhD, is a professor of pathology and laboratory medicine at the University of Wisconsin Medical School, Madison. He also is president of Westgard QC, Inc., (Madison, Wis.) which provides tools, technology, and training for laboratory quality management.