The Inevitability of Bias in Clinical Research

DrRich | September 23rd, 2010 - 6:47 pm

DrRich has said many times that clinical science is among the least exact of the sciences, and therefore, the results of clinical research are particularly susceptible to “spinning” by various interested parties, in order to yield the kind of results they would prefer to see.

Until recent times in American medicine, the parties who have been most interested in spinning clinical research have been the people who run drug companies and medical device companies (who need clinical research which supports the use of their products), and the medical specialists (who are more likely to be paid for performing medical procedures that are supported by clinical research). In writing about such data-spinning abuses, DrRich has particularly targeted his own Cardiology Guild, but only because he knows and loves cardiologists the best. He suspects that other specialists are doing exactly the same thing.

While DrRich has used reasonably gentle humor (laced, to be sure, with sarcasm and irony) to criticize doctors and their industry collaborators for twisting clinical data to their own ends, others have expressed the same concerns in much more indignant terms, and have threatened to employ professional sanctions, civil and criminal penalties, and everlasting perdition, to curtail such behaviors.

(Indeed, DrRich has always suspected that the real reason President Obama has not closed Guantanamo is so he has someplace to send recalcitrant American physicians who persist in accepting logo-ed plastic pens from drug reps, or who refuse to accept reduced Medicare/Medicaid reimbursement schedules, or who engage in the black market healthcare activities the President surely understands he is provoking. The one thing that can torpedo Obamacare completely is if American doctors refuse to go along, and any physician who shows signs of doing so will have to be dealt with harshly – if not by detention in exile, then by some other method.)

There is nothing wrong with a little old-fashioned American Puritanism, of course, and physicians and companies who behave badly ought to be punished. But DrRich begs his readers to understand that the inevitable bias in clinical research is not one-sided; it cuts both ways. And clearing the field, so that the only entities which are left to spin clinical research data will be the government-controlled expert panels, is a very bad idea.

DrRich must remind his readers that Obamacare provides for several distinguished expert panels, to be appointed by the executive branch of the federal government, to direct the studies, interpret the results, and apply the results to official reimbursement policies, of a species of clinical research which is called “comparative effectiveness research.”

Comparative effectiveness research comes in two flavors. First, there is the comparative effectiveness research whose unambiguous goal is to compare the clinical effectiveness among different treatment options, so as to offer physicians objective guidance in making the clinical decisions whose results are more likely to be clinically favorable to their patients. This kind of comparative effectiveness research is an unalloyed good, and it is as unassailable as babies and bunnies. Then there is Comparative Effectiveness Research (CER), which is to be operated by new government bureaucracies, whose agenda regarding what kind of effectiveness is actually to be compared is intentionally ambiguous, but which at the end of the day will be comparing cost effectiveness, as opposed to clinical effectiveness, so that doctors will make the clinical decisions whose results will be more favorable to healthcare cost containment.

Our policymakers have been studiously ambiguous about what they mean by  Comparative Effectiveness Research.

This ambiguity was made clear during the Obamacare debates when Peter Orszag testified on behalf of the administration before the Senate Finance Committee. When queried by skeptical Republicans on the ultimate goal of the proposed CER boards, Mr. Orszag was evasive. Specifically, when asked by Senator Kyle (R-Arizona) whether the CER board would be empowered to make decisions regarding which medical services will be reimbursed, Mr. Orszag finally replied, “Not at this point,” a reply which did not alleviate the suspicions of the minority party.

To state the ambiguity more plainly, it is clear that while the government’s CER panels will mainly be concerned about comparing cost effectiveness, the only kind of effectiveness they are willing to discuss publicly is clinical effectiveness. This studied ambiguity allows proponents of Obamacare to paint opponents of the CER panels as being against the “babies and bunnies” form of comparative effectiveness research, and thus reveal those nay-sayers as being beneath contempt, and unworthy of anyone’s attention. Meanwhile they will be free to advance their real “cost effectiveness” agenda.

Therein, of course, lies the government’s bias regarding clinical research. Since clinical research is the primary mechanism by which Obamacare proposes to cut the cost of healthcare, and since the new government panels provide the chief mechanism for controlling and applying the results of that clinical research, the government will be strongly biased toward research results that point toward the less expensive of the two treatments that are being compared.

The idea that government-controlled expert panels will be unbiased, of course, is so absurd that nobody can plausibly believe it. Where you go wrong, dear reader, is in believing that their bias can only take them so far; that clinical research, being science, will more or less yield the Truth; that while a biased party might shade things a bit in this direction or that, at the end of the day the answer which is reached will approximate the Answer.

In fact, clinical research is inherently biased, from the moment a research study is conceived. And those who conceive of, plan, conduct, and analyze the clinical study have every advantage. (This, indeed, is the very reason why everyone is so indignant about the studies conducted by medical industry and their minions in the medical academy.) That advantage is now, under law, defaulting to the government panels.

To be sure, many clinical researchers believe in their hearts and souls that bias can be eliminated through the use of randomized clinical trials (RCTs). In such trials, “like” groups of research subjects are divided randomly into two or more groups, and each group receives (for instance) a different therapy, whereupon differences in outcomes among the groups are attributed to the different therapies to which they were randomized. Indeed, the widespread belief that RCTs are the necessary and sufficient means to achieve “clinical truth” has become so deeply ingrained within the medical establishment that when anyone (such as DrRich) says otherwise, he immediately reveal himself as a scientific Neanderthal.

DrRich has previously observed that the widespread belief in RCTs has become like a Cult, whose creed can be reduced to three main tenets:

1) Data derived from randomized clinical trials represents Truth.
2) Data derived from non-randomized trials represents Falsity.
3) If you don’t believe this, you are a heathen.

Objective observers will find it at least a little ironic that an attempt to claim the scientific high ground has so obviously resulted in a new religion, replete with its own dogma.

The sad truth is that the results of RCTs are invariably dependent on the bias built into their design, and even if internally they are statistically legitimate, they can often send us down the wrong path.

Those who design RCTs (the smart ones, at least) know this. Like smart trial attorneys, they know the answer before they ever dare to ask the question. So they tailor their “question” in such a way as to yield the answer they want to get. Indeed, if a lawyer should end up asking a question in court that produces an unexpected answer, he or she is completely incompetent and ought to be sued for legal malpractice. In more cases than one might think, the same is true for those who design RCTs.

So, for instance, if you are a payer and want to limit the use of an expensive therapy, you design your RCT so that enrolled patients likely to respond to the therapy are diluted with a broad population of enrolled patients, many of whom are less likely to respond to the therapy, to assure that the average response of the whole population will be quite small. (In many instances the clinical characteristics of the likely responders and the likely non-responders will be reasonably apparent.)

On the other hand, if you are a company that wants to encourage the use of your expensive new product, you design an RCT that preferentially enrolls the relatively small subset of patients who are very likely to respond favorably, and then trust the marketplace (with a tweak from your DTC advertisements) to “extrapolate” the results to broader categories of individuals.

So RCTs do not eliminate statistical bias, as the dogma suggests. Rather, they simply offer an opportunity to control the statistical bias in your favor. Since most doctors (and most regulators, guideline writers, and reporters) don’t seem to get this, it becomes relatively easy to fool them.

What DrRich is saying, with regard to the government panels that will direct and interpret the CER (panels that will determine who gets what, when and how, and who gets paid for it and who doesn’t), is that even if those CER panels were not overtly biased against high-cost medical care, eliminating bias from their clinical research would be impossible. And given that the CER panels are being created expressly for the purpose of reducing high-cost medical care, the bias will likely become extravagant. But since that extravagant bias will be couched within the results of various RCTs, the Cult of Randomization will be invoked as “proof” that this expensive medical treatment is no better than that cheaper one.

DrRich has illustrated numerous times how the results of RCTs can be twisted and spun by interested parties, whether by private or government interests, to achieve the results one wants. The CER panels will (it seems obvious) become masterful at doing this.

The apparently widespread notion that industry-sponsored research is invariably biased, while government-sponsored research is entirely objective – and that therefore, the only thing we need to assure accurate clinical research is to have it all controlled by the government – is astoundingly naive.

DrRich believes that, since we cannot possibly eliminate bias from clinical research, we are more likely to approach the actual Truth if we: a) encourage clinical research by all parties – the government and private entities – so that, at least, we may be more likely to engender a “balance” of results; and b) insist that all clinical research be conducted with complete transparency, so that not only are the results made available to anyone who wants them, but also a complete accounting of all the other aspects of the research – including the study design, conduct of the study, and the analysis of the data.

Since bias cannot be eliminated from CER even if the federal CER panels wanted to (and they decidedly will not want to), then insisting on complete and total transparency (ideally, even to the point of making the raw data itself accessible), will be our chief defense. DrRich assumes, since covert rationing will undoubtedly be the CER board’s main, though unspoken, agenda, that such transparency will not be forthcoming without a fight.

Transparency will be worth fighting for, however. At least some bias in clinical research is unavoidable, so complete transparency is our best defense against the biased application of the results of clinical research, whether it is conducted by companies or CER panels.

4 Responses to “The Inevitability of Bias in Clinical Research”

  1. Urine says:

    If transparency is your question to bias, that does not leave a lot of hope :)

    There has been a lot of big talk about transparency, but with the complexity and lack of interest by the general public, real transparency may be just a nice theoretical idea.

  2. dmk says:

    Just getting to read this, directed from your Sermo appearance. I share your concerns about government appointing commissions to get the review that it wants, and “owning” the truth (“evidence based”, of course). Isn’t there an anology to be drawn between how these panels will pose questions and direct research and the “push poll” techniques used by politicians?

  3. I have been in clinical research for the past two decades. Bias in medical research is not envitable but preventable. As long as you keep the interest of the patient care and practice good clinical research practice with a open minded approach to address the protocol. Even before you start a research make your protocol does address all the ifs and buts.
    Drcjk

You can leave a response, or trackback from your own site.

Leave a Reply