The Inevitability of Bias in Clinical Research

Posted on May 11, 2009
Filed Under Comparative Effectiveness Research, General Rationing Issues |

Here’s a Podcast of this post:

 
icon for podpress  The Inevitability of Bias in Clinical Research [11:08m]: Play Now | Play in Popup | Download (316)

____________

After the last couple of posts, DrRich is now confident that: a) everyone agrees (even conservatives and Republicans), that comparative effectiveness research itself is a very good thing, and b) the controversy regarding CER has to do with how its results will be applied.

DrRich, of course, is certain that the power of CER (like every other potentially useful tool in medicine) will be corrupted toward the ends of covert healthcare rationing. As he has explained, the reason he is in favor of CER, as it is most likely to materialize under our current administration, is that under this arrangement the rationing will be particularly difficult to keep covert - and the result may be (DrRich prays) that a public discussion of healthcare rationing may at last become feasible, and perhaps unavoidable. A public discussion of healthcare rationing is the first and most necessary step toward any equitable system for controlling healthcare costs, and for making our healthcare system as fair and effective as possible.

Along the way toward expressing this opinion, DrRich asserted that comparative effectivness research - like any clinical research - is bound to be biased, and that, “The best you can do is to minimize the bias and illuminate that which remains, through complete transparency.”

This statement troubles Dr. Jan Krouwer, who (in addition to being a blogger) is a knowledgeable and deep thinker on matters related to the analysis of medical data - and in fact, he makes his living by providing systems for performing this kind of analysis. So DrRich takes his criticism seriously. To DrRich’s assertion, Dr. Krouwer replies, “But the goal of CER is to conclude which of several treatments is most effective. Bias tends to prevent valid conclusions so transparency has little value.”

DrRich suspects that Dr. Krouwer is more familiar with laboratory research than with clinical research. While in laboratory research the elimination of bias is always a challenging endeavor, DrRich believes that in clinical research it is an ultimately futile endeavor. And for this reason, anyone claiming to have performed unbiased clinical research is either blowing smoke up your nether regions, or is too ignorant about the realities of clinical research to be allowed to perform it. This is why, when reporting on clinical research, complete transparency - with, for instance, a detailed description of the methods of the research and a frank discussion of the limitations of the research - is so critically important.

To be sure, many clinical researchers believe in their hearts and souls that bias can be eliminated through the use of randomized clinical trials (RCTs). In such trials, “like” groups of research subjects are divided randomly into two or more groups, and each group receives (for instance) a different therapy, whereupon differences in outcomes among the groups are attributed to the different therapies to which they were randomized. Indeed, the widespread belief that RCTs are the necessary and sufficient means to achieve “clinical truth” has become so deeply ingrained within the medical establishment that when anyone (such as DrRich) says otherwise, they are immediately revealed as scientific Neanderthals.

Indeed, DrRich has previously observed that the widespread belief in RCTs has become like a Cult, whose creed can be reduced to three main tenets:

1) Data derived from randomized clinical trials represents Truth.
2) Data derived from non-randomized trials represents Falsity.
3) If you don’t believe this, you are a heathen.

Objective observers will find it at least a little ironic that an attempt to claim the scientific high ground has so obviously resulted in a new religion, replete with its own dogma.

The sad truth is that the results of RCTs are invariably dependent on the bias built into their design, and even if internally they are statistically legitimate, they can often send us down the wrong path.

Those who design RCTs (the smart ones, at least) know this. Like smart trial attorneys, they know the answer before they ever dare to ask the question. So they tailor their “question” in such a way as to yield the answer they want to get. Indeed, if a lawyer should end up asking a question in court that produces an unexpected answer, he or she is completely incompetent and ought to be sued for legal malpractice. In more cases than one might think, the same is true for those who design RCTs.

So, for instance, if you are a payer and want to limit the use of an expensive therapy, you design your RCT so that enrolled patients likely to respond to the therapy are diluted with a broad population of enrolled patients much less likely to respond, to assure that the average response of the whole population will be quite small. (In many instances the clinical characteristics of the likely responders and the likely non-responders will be reasonably apparent.)

On the other hand, if you are a company that wants to encourage the use of your expensive new product, you design an RCT that preferentially enrolls the relatively small subset of patients who are very likely to respond favorably, and then trust the marketplace (with a tweak from your DTC advertisements) to “extrapolate” the results to broader categories of individuals.

So RCTs do not in any way eliminate statistical bias, as the dogma suggests. Rather, they simply offer an opportunity to control the statistical bias in your favor. Since most doctors (and most regulators, guideline writers, and reporters) don’t seem to get this, it becomes relatively easy to fool them.

What DrRich is saying, with regard to the government board that will direct and interpret the CER (that will likely determine what gets paid for and what doesn’t), is that even if that federal CER board were not overtly biased against high-cost medical care, eliminating bias from their clinical research would be impossible. And given that the CER board indeed will be created expressly for the purpose of reducing high-cost medical care, the bias will likely become extravagant. But since that extravagant bias will be couched within the results of various RCTs, the Cult of Randomization will be invoked as “proof” that this expensive medical treatment is no better than that cheaper one.

DrRich has illustrated numerous times how the results of RCTs can be twisted and spun by interested parties, whether by private or government interests, to achieve the results one wants. The CER board will (it seems obvious) become a master at doing this. (The apparently widespread notion that industry-sponsored research is invariably biased, while government-sponsored research is entirely objective - and that therefore, the only thing we need to to assure accurate research is to have it all conducted by the government - is astoundingly naive. The imperative to covertly ration healthcare is every bit as strong among the payers of healthcare as is the profit motive among the sellers of medical products.)

Since bias cannot be eliminated from CER even if the federal CER board wanted to (and it decidedly will not want to), then insisting on complete and total transparency (ideally, even to the point of making the raw data itself accessible), will be our chief defense. DrRich assumes, since covert rationing will undoubtedly be the CER board’s main, though unspoken, agenda, that such transparency will not be forthcoming without a fight.

The greater the extent the CER board is successful in covert healthcare rationing (that is, in covertly withholding effective medical care from people who would benefit from it), and therefore the more people whose lives and health suffer from not having access to existing medical care, the more difficult it will become for the board to keep the details of their rationing activities hidden. This, DrRich hopes, will eventually result in a public debate on healthcare rationing.

In any case, whether all this amounts to a realistic possibility or mere wishful thinking, DrRich stands by his statement that at least some bias in clinical research is unavoidable, and for this reason transparancy is the best defense against the biased application of the results of clinical research.

Comments

6 Responses to “The Inevitability of Bias in Clinical Research”

  1. NG on May 12th, 2009 7:59 am

    How’s this idea for curtailing unnecessary and unproven treatments whether by design or just poor design. Set the significance level for results very high, such as 99.9%, and only pay for treatment that meet that level of significance. Maybe even add to this as a safeguard against any bias that the researcher/manufacturer has to pay a significant penalty for any actual clinical results that are subsequently proven to not follow the supposed 99.9 significance level!

    Too drastic you say? Well drastic problems require drastic solutions! You are painting a catch 22 riddle 9rightly so, IMO), so to solve that riddle takes some clout and pain!

  2. DrRich on May 12th, 2009 8:37 am

    NG,

    Your proposed solution is even better than you suggest. By holding all reimbursable medical treatments to the 99.9% efficacy test, we can immediately stop paying for (for instance) all cardiovascular therapies (bypass surgery, stents, statins, implantable defibrillators, pacemakers, etc., etc.) as well as all cancer treatments, antibiotics, and almost anything else in medicine.

    Fiscal crisis solved!

    Rich

  3. NG on May 12th, 2009 11:03 am

    Okay, let’s pack it up then and close it down.

    I love it when a plan comes together.

  4. Dr Dan on May 12th, 2009 12:24 pm

    So when ObamaCare gets up and rolling, and all that “expensive/unnecessary care” stops being paid for, will you consider that to be overt or covert rationing?

  5. Trusted.MD Network on May 12th, 2009 4:44 pm

    More on the three magic elements of reforming health care….

    To reform” health care” (and to fix the nation’s economy) we are told we must have electronic medical records, government funded and managed comparative effectiveness research and more prevention….

  6. DrRich on May 13th, 2009 8:35 am

    Dr Dan,

    Rationing healthcare is withholding at least some useful medical care from at least some of the people who would benefit from it. Rationing is covert when the entities conducting the rationing deny there’s any rationing going on, generally by insisting they’re actually doing something else entirely. For instance, Pay for Performance advances covert rationing by giving primary care doctors a checklist of approved activities they must perform during the 7.5 minutes they can spend with the patient, thus severely limiting their ability to perform any “non-approved” medical activities.

    One reason I support a federal CER board is that it will cause the covert rationing to be performed in a way that is relatively difficult to disguise as not-rationing. For instance, they will be reduced to declaring as “non-effective” (presumably by “guiding” the conduct of clinical trials, or by creative data analysis of completed trials) certain expensive medical treatments that are actually quite effective in at least some subset of the population. In other words, by trying to hide the effectiveness of expensive therapy, they will likely render the rationing much less covert. I’m guessing that interested patient groups who are being harmed by this more apparent rationing will cry “foul,” and potentially trigger an open discussion of healthcare rationing.

    So, to more directly answer your question, the rationing under the federal CER board will still be covert by design, but in practice will likely be much less so than the feds will want it to be.

    In saying all this I understand it’s not much to hang my hat on, but it’s the most direct way I can see to a system of honest, open healthcare rationing where the rules are completely transparent, and individuals can behave accordingly to further their own individual interests.

    Rich

Leave a Reply