Editor’s Note: To read another perspective on metrics and neurosurgery, read “Point: Clinical Registries for Neurosurgery.”

The current debate between randomized controlled trials (RCTs) and registries as the best way to advance our knowledge about neurosurgical practice is hardly a new one. Many neurosurgeons can still remember the reaction against RCTs that followed the EC-IC Bypass Trial, which published its negative results in 1985, followed immediately by a rash of negative comment in the neurosurgical literature. Stock went up in RCTs after NASCET and ACAS in the early 1990s, but down again after ISAT, up after positive trials on single metastasis resection, and down after SPORT. Most recently, the ARUBA trial has provoked a highly charged torrent of negative neurosurgical reaction against RCTs. Are RCTs really the best way to judge the value of a neurosurgical intervention, as the rest of medicine seems to be constantly lecturing us? Or are they really more like a dangerous toy that neurosurgeons shouldn’t be playing with, because we’re only going to get hurt?

Advantage of Randomizing
An RCT is a form of medical experiment in which outcomes in patients who do or do not receive a specific treatment are compared. Conceptually, every patient treated would be compared to another “control” patient who is perfectly identical in all respects, but this is clearly not possible. The reasons RCTs are expected to have less bias than other designs largely arise from the drawbacks of the other possible types of control population investigators can use instead of randomizing.

Historical controls typically are the weakest, because of the gradual shifts in diagnosis and treatment that inevitably take place over time. For example, after the chemosensitivity of oligodendrogliomas was first pointed out in the early 1990s, there was a dramatic shift in histopathological diagnosis of low-grade gliomas (already known to be subjective) away from astrocytomas and toward oligodendrogliomas in population-based databases — that is, the disease changed its definition over time. (11) During the same era, the baseline mortality of craniotomy for tumor in the U.S. showed a slow, steady decrease of about 50 percent over 12 years. (2) The changes in other endpoints, such as length of hospital stay, were even more dramatic. Studies that address quality of care that use historical controls often are affected by another bias called regression to the mean. This arises, for example, when a run of bad outcomes has prompted the study, such as shunt infections. A new treatment protocol has an unfair advantage if the time period immediately preceding the start of the new treatment was unusually bad because of a temporary, random increase in a low baseline rate.

The Registry Approach
Registry studies typically use concurrently treated patients as controls, because of the prospective nature of the data collection. Here, the biases can be more subtle. First, studies show that nonrandomized studies match the result of RCTs best when they are structured so that all patients in treatment and control cohort are equally eligible for the treatment. (3) This is difficult to guarantee in registries, not least because the factors that cause surgeons to choose one treatment over another are not known. Part of the selection process for many surgical treatments is overall good health, an impression that rests on intangibles that can be hard to quantify, even prospectively. In epidemiology, this gives rise to a well-known similar bias, the “healthy worker” phenomenon: When controls are selected from a working population rather than more generally, prognosis for endpoints like survival will be unexpectedly good. In general medicine, the longstanding impression that postmenopausal hormone replacement extended survival in women probably arose from this bias. Physicians were prescribing the hormones preferentially to healthier women. When RCTs were done, hormones actually had no beneficial effect on survival, and there were important toxicities. (1)

Neurosurgical Bias
An example of this bias in neurosurgery is the overall survival of patients after elective procedures, such as microvascular decompression, which is typically better than that seen in the overall population matched by age. This is because patients with multiple comorbidities get steered away from surgery and those who are generally healthy are steered toward it. How to capture this, or its opposite quality — “frailty” — in a registry is not immediately obvious. Other examples are well-known in which eligibility for a treatment is a strong predictor of better survival even in the absence of the treatment. In the 1990s, phase II trials showed substantially longer survival in glioblastoma patients treated with brachytherapy. Two subsequent RCTs showed no treatment benefit from brachytherapy. (10, 12) In retrospect, the eligibility requirements for brachytherapy treatment (such as performance status and volume) explained the survival difference. (7) This bias is called confounding by indication, and it is a common one. In this example, a registry could only adjust for eligibility by collecting, archiving and reviewing pretreatment images, which is expensive and time-consuming.

Some differences between cohorts can be adjusted away using statistical techniques, such as multivariate risk adjustment or propensity score models. The difficulty is knowing what factors need to be adjusted for. Here are some potentially surprising examples. First, in population-based studies, married patients with glioblastoma live 1.2 times longer than unmarried ones, a benefit not much smaller than that expected from chemotherapy. (4) Reasons may include earlier detection, stronger tendency to elect aggressive treatment, and better support during and after treatment that improves compliance. Second, travel distance to the clinic was a strong predictor of survival in phase II trials at a large academic medical center. (9) These trials largely test drugs later found to be effective against incurable cancer; patients who lived more than 15 miles from the center had one-third the death rate of those who lived closer. Clearly, social factors, such as aggressiveness in seeking out experimental treatment at a distant center, and having the performance status and income to make it possible, are at work here. Third, in many settings, adherence to an ineffective placebo proves to have a beneficial effect on a hard endpoint. For example, patients who adhere to placebo in fracture prevention trials have better bone density and fewer hip fractures (6); patients in myocardial infarction trials who adhere to placebo have less cardiac mortality. (5) Again, adherence to placebo must be a marker for other correlated “healthy” behaviors that actually explain the differences. If we were smart enough, we could collect all this data prospectively in our registry, and then adjust for it. Realistically, we will never be able to predict everything in advance (or we wouldn’t need to do the study). Randomization, in contrast, balances both known and unknown prognostic factors equally efficiently between treated and control populations.

RCTs vs. Registries
Problems with RCTs are well-known, and some are specific to surgical trials. Whether an RCT is ethical largely depends on equipoise. This important term is used differently by different authors, but broadly speaking, the relevant medical community and the individual physician taking care of the patient need to have a sufficient level of uncertainty as to whether treatment is better than control. This means admitting ignorance, which surgeons can be particularly reluctant to do. But if there is truly no variation in practice within the community (and hence no equipoise), a registry will not help because there will be no overlap between treated and untreated patients — so the comparison will be “apples to oranges.” Unlike drug trials, surgical RCTs face problems in defining and standardizing the intervention, and in assuring that trial surgeons can deliver it effectively and safely. Registries have the same problems, without the opportunities for pretrial training and surgeon certification that RCTs offer. When both arms in a trial are standard clinical practice, surgical RCTs may be affected by crossover that can complicate analysis. But every patient in a registry is assigned treatment through patient and physician preference, like a crossover in an RCT. Another important question is when to do the trial: too early and the new procedure may not have been perfected; too late, and the trial can be impossible because the procedure is already “standard practice.” Here a registry has an advantage over an RCT, if it is started early enough. RCTs are also very costly to conduct and time-consuming for trialists, probably more so than registries.

On the other hand, RCTs have a demonstrated capacity to change practice. Both NASCET and ACAS caused an increase in carotid surgery rates after results were publicly announced, even before final manuscripts were published. (8) After all, changing practice for the better is the purpose of medical research, and whether registry studies can “move the needle” still remains to be seen. When feasible, RCTs remain the gold standard in measuring the benefits of new medical and surgical treatments for all patients, including those needing neurosurgical care.

Frederick G. Barker II, MD, FAANS, is an associate professor of neurosurgery at Massachusetts General Hospital specializing in surgery for brain and skull base tumors. He currently serves as surgical vice-chair of the National Cancer Institute (NCI)-funded Alliance for Clinical Trials in Oncology, as a member of the NCI Brain Malignancies Steering Committee and the Coordinating Committee of the American Brain Tumor Consortium, and on the Neurosurgical Subcommittee of the NRG Oncology Group. Disclosure: Financial — The author receives consulting fees from the National Institutes of Health Cancer Treatment Evaluation Program.

References

1. Anderson GL, Limacher M, Assaf AR, Bassford T, Beresford SA, Black H, et al: Effects of conjugated equine estrogen in postmenopausal women with hysterectomy: the Women’s Health Initiative randomized controlled trial. JAMA 291:1701-1712, 2004.

2. Barker FG, 2nd, Curry WT, Jr., Carter BS: Surgery for primary supratentorial brain tumors in the United States, 1988 to 2000: the effect of provider caseload and centralization of care. Neuro-oncol 7:49-63, 2005.

3. Britton A, McKee M, Black N, McPherson K, Sanderson C, Bain C: Choosing between randomised and non-randomised studies: a systematic review. Health Technol Assess 2:i-iv, 1-124, 1998.

4. Chang SM, Barker FG, 2nd: Marital status, treatment, and survival in patients with glioblastoma multiforme. Cancer 104:1975-1984, 2005.

5. Coronary Drug Project Research Group: Influence of adherence to treatment and response of cholesterol on mortality in the coronary drug project. N Engl J Med 303:1038-1041, 1980.

6. Curtis JR, Delzell E, Chen L, Black D, Ensrud K, Judd S, et al: The relationship between bisphosphonate adherence and fracture: is it the behavior or the medication? Results from the placebo arm of the fracture intervention trial. J Bone Miner Res 26:683-688, 2011.

7. Florell RC, Macdonald DR, Irish WD, Bernstein M, Leibel SA, Gutin PH, et al: Selection bias, survival, and brachytherapy for glioma. J Neurosurg 76:179-183, 1992.

8. Gross CP, Steiner CA, Bass EB, Powe NR: Relation between prepublication release of clinical trial results and the practice of carotid endarterectomy. JAMA 284:2886-2893, 2000.

9. Lamont EB, Hayreh D, Pickett KE, Dignam JJ, List MA, Stenson KM, et al: Is patient travel distance associated with survival on phase II clinical trials in oncology? J Natl Cancer Inst 95:1370-1375, 2003.

10. Laperriere NJ, Leung PM, McKenzie S, Milosevic M, Wong S, Glen J, et al: Randomized study of brachytherapy in the initial management of patients with malignant astrocytoma. Int J Radiat Oncol Biol Phys 41:1005-1011, 1998.

11. McCarthy BJ, Propp JM, Davis FG, Burger PC: Time trends in oligodendroglial and astrocytic tumor incidence. Neuroepidemiology 30:34-44, 2008.

12. Selker RG, Shapiro WR, Burger P, Blackwood MS, Arena VC, Gilder JC, et al: The Brain Tumor Cooperative Group NIH Trial 87-01: a randomized comparison of surgery, external radiotherapy, and carmustine versus surgery, interstitial radiotherapy boost, external radiation therapy, and carmustine. Neurosurgery 51:343-355; discussion 355-347, 2002.