Fun With Randomized Trials, and Breasts

Posted on July 18, 2008
Filed Under General Rationing Issues |

Here’s a Podcast of this post:

 
icon for podpress  Fun With Randomized Trials, and Breasts [13:03m]: Play Now | Play in Popup | Download (197)

____________

The Cochrane Collaboration has created something of a stir with its latest review on the advisability of doing breast self-examinations, which concluded, in essence, that they’re not advisable. Specifically, they found that women who perform breast self-examinations end up producing more harm than good, so women should be discouraged from the practice.

The Cochrane Collaboration is a highly regarded resource for those who value evidence-based medicine, medical guidelines, and the randomized clinical trials that support such tools. The experts at Cochrane conduct reviews of all randomized clinical trials (RCTs) that meet their strict criteria for scientific and statistical robustness, and periodically publish summaries of the scientific evidence thereby derived, on particular clinical topics. Cochrane reviews are thorough and straightforward, and virtually always present an accurate reflection of the up-to-date evidence as supported by RCTs.

Despite the respect in which the Cochrane Collaboration is held, DrRich finds it at least mildly interesting that their conclusions regarding breast self-examination have not been generally construed as being politically incorrect. After all, several aspects of political incorrectness, ones at least as incorrect as your more standard species of political incorrectness, can be readily identified within the Cochrane review. For example, one could easily allege that anti-women, anti-self-empowerment, or anti-early-detection thinking riddles this report. And if the review had emanated from say, the White House, one almost certainly would. But in fact most media reports, while expressing a certain amount of surprise, have seemed very reluctant to criticize the Cochrane Collaboration’s conclusions. And physician experts who have been excavated by the media for their comments have also given at least tepid support.

(Not all commentators have blindly accepted Cochrane’s recommendations. DrRich refers you to his colleague Trisha Torrey, for example, who expresses an appropriate amount of skepticism. But the traditional media, and their Rolodex physicians, have on the whole swallowed it. See here and here.)

The general support for Cochrane’s report on breast self-examination, DrRich submits, is a direct reflection of the exulted position that RCTs have achieved today all across the modern medical (and media) landscape. If a statistically legitimate RCT reaches some conclusion, no matter how strange, counterintuitive, or wrong that conclusion might seem, it is a conclusion that must be accepted. That this belief in RCTs trumps even the powerful social force of political correctness speaks to its ultimate strength.

DrRich has previously observed that our widespread belief in RCTs can be reduced to three main tenets:

1) Data derived from randomized clinical trials represents Truth.
2) Data derived from non-randomized trials represents Falsity.
3) If you don’t believe this, you are a heathen.

Objective observers will find it at least a little ironic that an attempt to claim the scientific high ground has so obviously resulted in a new religion, replete with its own dogma. True Believers will not see the irony, thus providing even more evidence that what we’re dealing with here is indeed a religion and not objective scientific thought. (Religions deal in mystery, and not irony.)

The sad truth is that the results of RCTs are invariably dependent on the bias built into their design, and even if internally they are statistically legitimate, they can often send us down the wrong path.

Those who design RCTs (the smart ones, at least) know this. They are like smart trial attorneys, in that they know the answer before they ever dare to ask the question. So they tailor their “question” in such a way as to yield the answer they want to get. Indeed, if a lawyer should end up asking a question that produces an unexpected answer, he or she is completely incompetent and ought to be sued for legal malpractice. In more cases than one might think, the same is true for those who design RCTs.

So, for instance, if you are a payer and want to limit the use of an expensive therapy, you design your RCT so that enrolled patients likely to respond to the therapy are diluted with lots of enrolled patients much less likely to respond, to assure that the average response of the whole population will be quite small. (In many instances the clinical characteristics of the likely responders and the likely non-responders will be reasonably apparent.)

On the other hand, if you are a company that wants to encourage the use of your product, you design an RCT that preferentially enrolls patients who are very likely to respond favorably, and then trust the marketplace (with a tweak from your DTC advertisements) to “extrapolate” the results to broader categories of individuals.

So RCTs do not in any way eliminate statistical bias, as most seem to think. Rather, they simply offer an opportunity to control the statistical bias in your favor. Since most doctors (and most regulators, guideline writers, and reporters) don’t seem to get this, it becomes relatively easy to fool them.

DrRich does not know if the people who designed the RCTs looking at breast self-examination tried intentionally to bias the results against self-examination, or if it was an accident. But that is what they did.

The RCTs which the Cochrane Collaboration reviewed looking at breast self-examination indicate that, in large populations of women in Russia and China, who go on to receive Russian and Chinese healthcare, breast self examination did not improve overall survival. And since those who did self-examination underwent twice as many breast biopsies, many of which revealed benign lumps, they experienced net harm. DrRich does not quibble with any of this. It is almost certainly true.

On the other hand, while most RCTs do not reveal it (since they look at aggregate results and not individuals), it is most often the case that some individual participants in even a negative study will experience benefit from the intervention being tested. In the breast-self examination studies, for instance, it is a certainty that individual women benefited at least to the extent that their breast cancers were detected earlier than they otherwise would have been. (In the Russian study this was proven to be the case.) These women had at least a shot at better survival by virtue of their earlier detection.

But whatever the overall results of these RCTs conducted in such exotic locales, they can have nothing whatsoever to do with women in America in 2008, who receive far more aggressive, tailored, and sophisticated therapy for breast cancer than women in virtually any other country in the world. (The optimal treatment of breast cancer depends on correct staging, on correct genetic testing of the tumor, and on optimizing the individual’s surgical and medical therapies, often employing very new drugs.) In any case, earlier detection of breast cancer is far more likely to be a significant factor in determining outcome in the U.S than it is in recently (or presently) Communist countries with 3rd world healthcare systems.

To be sure, DrRich does not pretend to know anything about the various specific institutions in Russia and China where these studies were conducted. Perhaps they were conducted in those rare showcase institutions that appear here and there under authoritarian regimes, of the sort Michael Moore frequents when he goes to Cuba, and where the level of medical care is said to more nearly approximate medical care in the United States. (Occasional institutions like this will survive even in the U. S. after Healthcare Reform, so that our Congresspersons will have someplace to go.) While this seems very unlikely - could a few showcase hospitals really accommodate the nearly 400,000 women enrolled in these studies? - one must suppose it is possible.

But even if all 400,000 study participants received showcase healthcare, advances in the therapy of breast cancer since the 1990s, when these women were studied, would likely yield different (and better) results today. Does any American oncologist believe that the early detection of breast cancer in 2008 does not improve a woman’s prognosis?

In light of such considerations, why are American doctors apparently so ready to accept the results of the Cochrane review, and to stop recommending breast self-examination? The answer is clear: It is because the data came from RCTs, and since RCTs always yield the truth (you heathen), all other considerations must fall away. Such is the depth of our faith.

DrRich will leave it for another day to discuss the favorable implications to the healthcare system of discouraging breast self-examination (such as having to pay for fewer biopsies), or the lessons that ought to be learned from this example about the advisability of blindly accepting formal guidelines just because they are derived from RCTs, and instead will simply give some friendly advice to the women who might be reading this blog:

First, it is probably a good idea to continue with your breast self-examinations. The downside of doing so is that you may be subjected to a breast biopsy for a benign lump. So take that into consideration. But the upside is that self-examination offers the (proven, according to the RCT data) potential for earlier detection, and therefore offers better odds of long-term survival if cancer is present.

And second, if breast cancer is diagnosed, try to get your therapy in the U.S. rather than in Russia or China. For the average patient in those countries, it would appear that cancer therapy sucks.

Finally, DrRich will close with this plea: Can our guideline writers, at least in the U.S., please refrain from creating new guidelines that are suitable for 3rd world healthcare systems, at least for the year or so it might take for the American healthcare system to actually be reduced to those levels?

Comments

10 Responses to “Fun With Randomized Trials, and Breasts”

  1. James Gaulte on July 18th, 2008 8:58 am

    I remember blogging once about RCTs and suggested that there was the possibility of rigging the game as well as cooking the books and was accused of being a drama queen.And tangentially I quote again the comments of one of your fellow cardiologists, Tom Giles, who seems to enjoy pointing out that some of the RCT zealots might like to volunteer for a RCT to test the hypothesis that absence of parachutes is a risk factor for death for those who jump from airplanes.

  2. Dr. Val on July 18th, 2008 10:15 am

    I might also add that the cost of this screening test makes the ROI (in early intervention for breast cancer) all the more reasonable. Nice post, Dr. Rich. Well written and entertaining as usual. :)

    Of course, the alties will come out of the woodwork and point to your arguments as “proof” that evidence-based medicine is fundamentally flawed and that personal instincts and “the wisdom of ancients” is the way to go. Sigh.

  3. DrRich on July 19th, 2008 7:34 am

    James,

    True Believers would reply that they’d be happy to recruit patients for such a study (purveyors of RCTs never volunteer themeselves to be randomized to anything), but that there’s simply no funding for it.

    Rich

  4. DrRich on July 19th, 2008 7:44 am

    Val,

    And the irony is that both the Alties and the RCT Cultists will have reached their apparently opposite positions by using the same kind of Magical Thinking, since (to them) the act of randomization seems to impart exactly the same kind of mystical power to the process as chelation, the energy of touch, or water memory.

    The difference is that RCTs have tremendous potential to help us tease out the facts, but only if used logically and critically, with a clear-eyed respect for the method’s limitations.

    Rich

  5. Tom on July 22nd, 2008 1:39 pm

    Because of her self-examination, my wife was diagnosed with breast cancer three months earlier than it would have been had it been diagnosed from her annual OBG exam and mammogram. It might have been diagnosed earlier if she had done the self-exam more regularly.

    Will her outcome be better because of the three month gain? Very hard to know, probably not, but we’re still glad she got it. She is doing fine five years post end of lumpectomy, radiation, and chemo, with anastrozole after.

  6. AnnR on July 22nd, 2008 9:06 pm

    Thank you for your take on where this seemingly nutty recommendation came from!

    I don’t know any American women who would think, upon feeling an honest-to-god lump, that it didn’t warrant a medical look-see to eliminate the possibility that it was something dangerous.

  7. John Beaty on July 25th, 2008 4:14 pm

    Hi DrRich

    Can you discuss the issues around early detection not extending survival? By which I mean, if you ave cancer that previously killed you in 90 days, and the new detection finds it 90 days earlier, but your survival time is now 180 days, so you really haven’t gained anything.

    Also, I’d love to hear some talk about surrogate endpoints.

    Thanks,

    john

  8. DrRich on July 25th, 2008 4:34 pm

    John,

    I am a simple country cardiologist, and am not an expert in cancer care, especially breast cancer care which, if done correctly, requires very sophisticated decision-making. My post has to do with the misinterpretation of RCTs, and not so much the specifics of breast cancer treatment.

    My point (again) is that the relatively primitive therapy that women almost certainly received in Russia and China in the 1990s would seem to have little bearing on the treatment women in America receive in 2008, no matter what the endpoints might have been. So taking the results of such clinical trials at face value, and as a result advising American women to stop doing an inexpensive early detection technique is, on its face, absurd. Yet it is being done, because RCTs have acquired a mystical value.

    A discussion of appropriate methods and techniques and endpoints for doing a useful breast cancer RCT is beyond my feeble abilities, I’m afraid.

    Rich

  9. Toxic MD on August 6th, 2008 12:28 am

    Another aspect is the sometimes difficult nature of getting negative trials published. Maybe they just aren’t sexy enough but authors seem to be greatly rewarded for coming to positive conclusions and naturally design their studies to reach them.

  10. Classic Covert Rationing Podcasts : The Covert Rationing Blog on January 26th, 2009 8:54 am

    [...] Fun With Randomized Trials, and Breasts July, 2008 [...]

Leave a Reply