METHOD VALIDATION -
SELECTING A METHOD TO VALIDATE
James O. Westgard, Ph.D.

• Establishing a laboratory testing process
• Method characteristics
•     Application characteristics
•     Methodology characteristics
•     Performance characteristics
• AST example from the literature
• Cholesterol example
•     General characteristics
•     High volume automated laboratory
•     Point-of-care
• Method evaluation vs method validation
• References

Error assessment is what method validation is about, as discussed earlier in MV - The Inner, Hidden, Deeper, Secret Meaning. However, before getting to the assessment of errors, you have to first select the method to be validated. Method selection is a different process that needs to be understood in relation to the validation process that will follow. In fact, there are several other processes that are essential for establishing a routine method of analysis.

Establishing a laboratory testing process

Important activities for establishing a routine method of analysis are shown in the accompanying figure. The blocks at the bottom illustrate the key steps involved in routine analysis, where the laboratory acquires specimens, performs tests, checks statistical QC, and reports test results. Those activities are generally regarded as the real work of the laboratory.

However, for analysis to become routine, the other activities shown in the figure are very important. The selection of the diagnostic test is actually the first step, but this is often skipped for common tests whose medical usefulness is well accepted. For these well accepted tests, we usually start with the selection of the method (the box with the darkest border), then validate its performance. If performance is acceptable, the method is implemented for routine service. If performance is not acceptable, the laboratory may develop some improvements, although that is becoming increasingly difficult with the high state of automation of many analytical systems. Today it's more likely that a laboratory would select another method rather than attempt to make improvements, then start the validation process over again for the new method.

Once a method is demonstrated to perform acceptably, the method must be implemented for routine operation. This involves defining the standard operating procedures and documenting the procedure, selecting an appropriate QC procedure for monitoring routine performance, and training personnel to operate the new method. While in routine service, problems will undoubtedly be identified through QC, which will lead to preventive maintenance procedures to minimize or eliminate those problems. Routine operation is often the simplest part of this overall process if the laboratory does a good job of selecting the method, validates method performance, implements the method through careful and thorough in-service training, monitors method performance with a QC procedure that has a low false rejection rate and appropriate error detection, and aggressively maintains the method to identify problems, eliminate sources of error, and prevent future problems from occurring.

Method characteristics

The aim when selecting a method is to choose the method that has the best chance of achieving the laboratory's service requirements. The process of selection consists of defining those requirements, searching the technical literature to survey information about available methods, then selecting the method whose characteristics best satisfy the laboratory's service requirements.

Careful definition of the requirements is essential [1]. If this step is overlooked, the laboratory may spend considerable time and effort evaluating a method that will not be satisfactory, even if method performance is okay. An example might be a point-of-care method that turns out to be too expensive even though its analytical performance is acceptable. Cost should be considered out-front when selecting the method to be evaluated, not after validating method performance.

What characteristics of a method are important? In general, they can be divided into three categories.

• Application characteristics are factors that determine whether a method can be implemented in a particular laboratory situation. They consist of cost-per-test, types of specimens that can be analyzed, sample volume, turnaround time, workload, equipment and personnel requirements, space, portability, and safety considerations.
• Methodology characteristics are factors which, in principle, should contribute to best performance. In general, these are concerned with the analytical sensitivity and analytical specificity of the method of analysis. They consider the choice of chemical reaction, optimization of reaction conditions, principles of standardization and calibration, and the rigor of the analytical procedure.
• Performance characteristics are factors which, in practice, demonstrate how well a method performs. These include the working range, precision, recovery, interference, accuracy, and sometimes detection limit.

Let's start with a non-laboratory example to illustrate these different kinds of characteristics. Assume you're selecting a new motorized vehicle.

• Application characteristics would include your expected use, type of vehicle, price range or maximum cost, body style, color, etc. Your expected use with help define the type of vehicle - tractor, truck, car, motorbike. Assuming it's a vehicle that will be used to go to work, let's look for a car. If your favorite color is red, you don't have to try out a black car - you can tell by looking whether it satisfies this application characteristic.
• Methodology characteristics can be illustrated by the following. If you live in Wisconsin, or worse yet Minnesota (no offense meant - I've actually lived there and still visit often), you might want a car with good traction for the winter weather. A four-wheel drive vehicle would be expected to perform better than two-wheel drive. If good mileage were also important, a four-cylinder engine would be a better characteristic than a six or eight cylinder engine. Both four-wheel drive and number of cylinders (or engine size) are methodology characteristics that should help satisfy your performance needs.
• Performance characteristics are those factors that you can evaluate by driving the vehicle. Standard measures of performance are acceleration, handling, ride, braking, and mileage. Special measures might include off-road or winter-weather traction. You don't test-drive an expensive, large, black, eight-cylinder, two-wheel drive truck because that vehicle doesn't satisfy your application and methodology characteristics.

AST example from the literature

A detailed list of characteristics for analytical methods which measure the activity of aspartate aminotransferase (AST) has been published [2] and provides a good example of the kinds of factors to consider.

• Application characteristics include cost/test, specimen types, specimen volumes, time of analysis, workload and rate of analysis, run size, equipment required, personnel required, efficiency, and safety.
• Methodology characteristics include analytical sensitivity, analytical specificity, measurement reaction, temperature, buffer type, buffer molality, pH, alpha-ketoglutarate concentration, aspartate concentration, NADH concentration, MDH activity and purity, MDH preparation, LDH addition, pyridoxal phosphate addition, blanks, mode of measurement, time period of measurement, and standardization.
• Performance characteristics include within-run imprecision, working range, substrate depletion check, pyruvate interference, acetoacetate interference, ammonia interference, bilirubin interference, turbidity interference, NaF/oxalate interference, pyridoxal phosphate activation, total or day-to-day imprecision, comparison with a selected method of reference, expected reference interval, timing, temperature control, stability of reagents, stability of reaction product, pipetting errors, and spectrophotometric conditions.

Cholesterol example

Cholesterol will be used as an example test throughout this discussion of method validation. Cholesterol tests are performed in a variety of settings - from central laboratories to point-of-care applications to health fairs in the shopping mall. In the USA, the National Cholesterol Education Program (NCEP) provides guidelines for the use and interpretation of cholesterol tests, and also defines the desired method performance as an allowable bias up to 3.0% and an allowable coefficient of variation (CV) up to 3.0%. Cholesterol is also one of the analytes regulated by the Clinical Laboratory Improvement Amendments (CLIA), which provides proficiency testing criteria for acceptable performance. Thus, there are official USA requirements for quality that are defined on a national basis. In many ways, cholesterol provides a model system for understanding how to manage the analytical quality of a laboratory test [3].

General characteristics to be considered are the following:

• Application characteristics should include type of specimen, volume of specimen, workload appropriate for the testing situation (high volume centralized lab vs low volume point-of-care), cost per test, time for analysis, operator skills, and operator training requirements.
• Methodology characteristics should include type of standards or calibrators, traceability of standard or calibrator assigned values, chemical principle, reagents and reaction conditions, measurement principle, and measurement capabilities.
• Performance characteristics should include working range, imprecision, bias vs the Abell-Kendahl reference method [4], interference, and total errors less than 10%.

High volume automated laboratory characteristics can be defined more specifically:

• Application characteristics are serum samples, sample size of less than 10 microliters, test throughput of 200 per hour, cost less than $0.25 per "click" or $0.50 per test.
• Methodology characteristics are enzymatic cholesterol esterase reaction, calibrators traceable to the national Abell-Kendahl reference method [4], and accuracy verified by comparison of results with a reference lipid laboratory.
• Performance characteristics are a working range from 50 to 500 mg/dL, allowable total error of 10% at a concentration of 200 mg/dL, method CV or 3% or less, method bias of 3% or less, no interference with up to 10 mg/dL bilirubin, no interference with lipemia or up to 600 mg/dL triglycerides, and minimal interference from hemolysis.

Point-of-care (POC) characteristics might be quite different, particularly the application characteristics that must take into account for the POC setting:

• Application characteristics are whole blood specimen from fingerstick or heparinized sample, turnaround time of 10 minutes or less, ease of use for operators who are not trained in laboratory testing, minimal maintenance and downtime, works or doesn't work operation, cost less than $4.00 per test, impossible to use non-calibrated reagent lot, small, lightweight, and portable.
• Methodology characteristics are enzymatic cholesterol esterase reaction, calibration built into test system by manufacturer with lot specific values traceable to Abell-Kendahl reference method [4], and accuracy verified by comparison of results with a reference lipid laboratory.
• Performance characteristics are a working range from 100 to 400 mg/dL, allowable total error of 10% at a concentration of 200 mg/dL, no interference with hematocrits up to 55%, no interference from bilirubin up to 5 mg/dL, no interference from lipemia up to 600 mg/dL triglycerides, and minimal interference from hemolysis.

Method evaluation vs method validation

The difference in the number of characteristics and the detail in the AST and cholesterol examples reflects the effort needed to establish or evaluate method characteristics of an untested method (AST example) compared to the effort needed to verify or validate the performance of a well tested method (cholesterol example). Most moderately complex methods have been well studied by manufacturers as part of their own development process, therefore, the laboratory can perform less extensive studies to validate method performance. Highly complex methods should be studied more thoroughly. Any methods that are modified or developed by the laboratory itself must be evaluated extensively.

References:

1. Westgard JO, deVos DJ, Hunt MR, Quam EF, Carey RN, Garber CC. Concepts and practices in the evaluation of laboratory methods. I. Background and Approach. Am J Med Technol 1978;44:290-300.
2. Westgard JO. Precision and accuracy: Concepts and assessment by method evaluation testing. CRC Critical Reviews in Clinical Laboratory Sciences 1981;13:282-330.
3. Wiebe DA, Westgard JO. Cholesterol - a model system to relate medical needs with analytical performance. Clin Chem 1993;39:1504-1513.
4. Cooper GR, Smith SJ, Duncan IW, et al. Interlaboratory testing of the transferability of a candidate reference method for total cholesterol in serum. Clin Chem 1986;32:921-929.