Skepfeeds-The Best Skeptical blogs of the day

Case study : Bad Studies

Posted in Skepdude by Skepdude on April 13, 2009

Homeopathy enthusiasts are pointing to yet another bad study as proof that homeopathy has a basis in science. This one is from March 2007 and is titled “Homeopathic and conventional treatment for acute respiratory and ear complaints: A comparative study on outcome in the primary care setting“. It purports to show that homeopathic remedies worked just as well as conventional remedies to treat “accute runny nose, sore throat, ear pain, sinus pain or cough”. You can read the details at the link above. What I want to concentrate here is on just how badly designed this study is. It seems to me as though it was set up specifically to produce the sort of answer that the homeopaths were after. Nevertheless let me present my arguments and you can make up your own mind.

Before we look at this specific study, let us go over the basic things to look at when reviewing any study. How well the study is designed and how well it conforms to certain rules has a profound effect on how much reliance we can put in its results. One way of assesing how good a study is, is what is known as the Jadad Scale. The Jadad scale is a simple checklist that helps you decide how well designed, therefore how trustworthy a clinical study is. It concentrates on the following areas: Randomizatin, Double Bliding, Withdrawals and Drop Outs. Here is a typical way of calculating the Jadad Score.

Basic Jadad Score is assessed based on the answer to the following 5 questions.

The maximum score is 5.

Question Yes No

1. Was the study described as random? 1 0

2. Was the randomization scheme described and appropriate? 1 0

3. Was the study described as double-blind? 1 0

4. Was the method of double blinding appropriate? (Were both the patient and the assessor appropriately blinded?) 1 0

5. Was there a description of dropouts and withdrawals? 1 0

Quality Assessment Based on Jadad Score

Range of Score Quality

0–2 Low

3–5 High

So let us go over the homeopath’s study and see how it ranks based on the Jadad Score.

1-Was the study described as random? NO – 0 points (cumulative)


The study was designed as an international, multi-centre, comparative cohort study of non-randomised design.

Not only that , but the patients were asked which group they wanted to be in, homeopathy or conventional medicine (misspelling of the word enrollment is theirs not mine! Also emphasis is added by me).

Upon enrolment in the study, patients, or the patients’ legal guardians were asked for their treatment preference. In the homeopathy group, 81% of patients had a preference for homeopathy, 18% had no treatment preference. In the conventional group, 55% of the patients’ preferred conventional treatment, 2% homeopathy and 43% had no treatment preference.

Fun Fact – 81% of the patients in the homeopathy group had chosen homeopathy and the results from the homeopathy group were…drum roll….86.9% reported complete recovery. Can you say placebo?

2-Was the randomization scheme described and appropriate? There was none. – 0 points (cumulative)

Randomization is very important when setting up clinical studies. Not only is it important to randomize the patients, but also how you randomize them matters. Different methods of randomization rank higher than others. According to Wikipedia:

Randomisation is a process to remove potential distortion of statistical results arising from the manner in which the trial is conducted, in particular in the selection of subjects. Studies have indicated, for example, that nonrandomised trials are more likely to show a positive result for a new treatment than for an established conventional one.

I haven’t checked that claim on the last sentence, so take it with a grain of salt, even though it does make sense.

3-Was the study described as double-blind? No, there was no blinding whatsoever, doctors knew what treatement each patient was getting and patients knew it too (they got to choose remember) – 0 points (cumulative)

Since it was not possible to blind patients for their treatment, potential reporting bias from patient’s expectations may have influenced the outcome.

You think?

4-Was the method of double blinding appropriate? (Were both the patient and the assessor appropriately blinded?) There was no double blinding, there wasn’t even single blinding. – 0 points (cumulative)

This is where I would start worrying if I was trying to use this study to prove my point. We’re up to question 4 of 5 and they have 0 points!

5-Was there a description of dropouts and withdrawals? Not even close, they only mention that 6 people who got no treatment were dropped. – 0 points TOTAL!

This study ranks as possibly the worst designed study you could come up with. There was no randomization, no blinding of any sort, let alone double, no control group, in other words nothing that would lend it even a slight amount of legitimacy. The authors seem to have benn aware of this, for they make sure to make the following point: None of that stuff really matters, our study is good enough as it is! Notes in red are my comments.

Objective data collection and evaluation is needed to assist physicians in patient care and advance the quality of medical practice [2] This study will presumably be objective!. Clinical trials, especially randomised controlled trials (RCTs), are generally accepted as producing the highest level of evidence for medical interventions. I feel there’s a “but” coming! Driven by the discovery of new pharmaceutical substances, demands from regulatory authorities for clinical data and the need of physicians for evidence based treatment strategies, the methodology of RCTs became the subject of research itself. Within this context, the strengths and weaknesses of such trials have been debated [3]. Placebo-controlled RCTs are indispensable for the development of pharmaceutical agents with unknown efficacy and safety profiles Such as maybe homeopathic agents. On the other hand if the efficacy and safety of an agent is known why would one even bother to do a study?. Their limitations result from highly standardized study protocols and patient populations, which may create artificial situations that differ from daily practice Oh, I see they are more tightly controlled and have stricter requirements, and THAT makes them problematic. What? . Moreover, even the fact that patients are enrolled into a placebo-controlled clinical trial will influence treatment outcome, sometimes leading to high placebo or low verum response rates [4] Somehow I did not think it was a matter of high or low, I thought it was a matter of the truest measure which is the point of the control groups. Further, proper blinding should guarantee the truest results possible. Consequently, more practice-based studies have been developed such as pragmatic RCT’s or non-randomised cohort studies. In other words, when you can’t live up to these standards make up more lax standards and claim they are just as good. Pathetic! Especially non-interventional outcomes studies have only few inclusion and exclusion criteria. Therefore they may provide information about a broad and heterogenous patient population thus resulting in high external validity for daily medical practice Actually lack of controls will result in exactly the opposite, it will be useless for daily medical practice. It may provide a good gauge for people’s ability to deceive themselves though. However, the fact that patients are not randomly assigned to treatments in such outcome studies may lead to baseline differences between groups and makes the interpretation of the results more susceptible to bias. May? That’s putting it mildly! This disadvantage may be overcome, at least in part, by the application of statistical methods to control for baseline differences between treatment groups No it can’t, otherwise randomization would not be required, EVER. Good statistics can never make up for bad data. Statistics rely on the data itsel. The above claim makes no sense!

Fun Fact –

Apart from the ongoing discussion about clinical evidence, complementary therapies are well integrated into primary care in most Western countries

Yeah appart from the fact that CAM has not been shown to work, IT IS POPULAR. Good enough for me!


This study is horrendously designed. It lacks all of the basic requirements that every clinical trial should have, such as randomization, double blinding, control group etc.  Based on that fact alone, regardless of the sample size, regardless of how careful and precise the statistics, the results of such study will be completely unreliable. The data set is corrupted due to the lack of controls, as such it does not matter how carefully you analyze it, the result would be meaningless. Even if it had told us that homeopathy is useless, we would still have to ignore it. And ignore it, I, we and all the science based community will. Sorry homeopaths, you’re still stuck at 0. Good luck next time.

15 Responses

Subscribe to comments with RSS.

  1. Rightside said, on April 14, 2009 at 8:55 AM

    I’m sure that the original trials of Vioxx passed your checklist as a “good study”. And that produced a drug that has injured thousands of people and killed many. So much for your “scientific method”.

    • Skepdude said, on April 14, 2009 at 9:14 AM

      Irrelevant to what I am saying. Do you have any comments on my actual entry?

  2. Rightside said, on April 14, 2009 at 10:05 AM

    Absolutely relevant to the entry. The whole basis of your “scientific method” is a corrupt fantasy when you don’t see that the maiming and killing of people with it is relevant.

    • Skepdude said, on April 14, 2009 at 11:22 AM

      Really? Are you aware that there are about 40,000 car accident related deaths per year in the US alone? What conclusion do you draw from that? The scientific method is the best tool we have to evaluate claims, otherwise we’re back to bloodletting. The scientific method is responsible for you being able to post a comment on this entry, via the computer, via the internet.

      It is always amusing to see people use the products of the scientific method on their daily life, and turn around and accuse the scientific method of being a fantasy. Do you not see the hypocrisy?

      Go ahead accuse me of being close minded!

  3. Rightside said, on April 14, 2009 at 11:45 AM

    Yes, shit happens. So what? And a computer is very different than the human body. It is interesting that you confuse the two.

    You extol a method which you call a scientific method of determining efficacy and safety yet it allows unsafe drugs to the market??? You smugly reconcile this fact with the fact that shit happens. This doesn’t look like a sound scientific principle but a skeptic magical thinking principle.

    • Skepdude said, on April 14, 2009 at 11:59 AM

      Ah so you expect perfection! Are you aware of the False Dichotomy fallacy, because that is what you are engaging in here. What you are basically saying is that either it works perfectly all the time or it doesn’t work at all, and if you can find an instance when it doesn’t that means it is unreliable.

      Well then, I challenge you to give me an example of any enterprise engaged in by human beings that is perfect 100% of the time. Just one example!

      PS: yes shit happens, it always has and it appears it always will, and if you live on planet Earth that is something you have to accept, regardless if you like that or not. Yes I do extol the scientific method. You tell me what is better at finding the truth and I will extol that. Until you do that you’re in no position to criticize me!

      I do not confuse a computer with the human body. I say that the principles of the scientific method are the same and have been used repeatedly to give us tangible results such as computers, and cars, and medicine, and airplanes and telephones and everything else that makes 21 century life so comfortable and enjoyable.

      There is no magical thinking involved in accepting that the scientific method cannot guarantee 100 % accurate results. It is being used by humans after all who are fallible/corruptible. At least we, the pro-science crowd, can see and accept our shortcomings? Can you?

  4. Rightside said, on April 14, 2009 at 12:22 PM

    I’m not the one presenting a specific “scientific method” as the only and perfect way of vetting drugs when it really doesn’t. This is magical thinking.

    When the outcome is so poor then why do you keep harping on it or even writing about it.

    “Pro-science Crowd”???
    You can smugly ridiculing the millions of people who use homeopathy. With knowing only a small amount about them, you can claim they are morons and unscientific but it doesn’t explain the many who are well educated and have good scientific backgrounds like myself and friends who use it as well as the many Medical Doctors who practice and study it.

    • Skepdude said, on April 14, 2009 at 1:15 PM

      “I’m not the one presenting a specific “scientific method” as the only and perfect way of vetting drugs”-Straw Man Fallacy-I never claimed that, in fact I specifically said it is not perfect. Is English not your first language?

      “When the outcome is so poor then why do you keep harping on it or even writing about it.” – Straw Man – I never said that the outcome is so poor, I reiterate that it is not perfect. Is English not your first language?

      “the many who are well educated and have good scientific backgrounds like myself” – Ridiculous – You can’t have a “good scientific background” and claim that the scientific method is a “corrupt fantasy”, or you risk coming across as a raving lunatic. That’s kinda like saying you’re a Christian but you don’t believe in God!

      By the way where is the one example of a 100% perfect human enterprise I asked you about? I am addressing your points, why do you not address mine? Answer the questions that are asked to you directly, don’t start assuming you know how or what I think, you’re embarrassing yourself.

  5. Rightside said, on April 14, 2009 at 1:39 PM

    Sorry- multi tasking and thus poor engelish, spelling and grammar.

    I am talking about your application of the scientific method. You seem to think you are the perfect scientist when it appears to me that you are simply a magician like the guy in the sidebar- using science to support your preconceived skeptic ideas in order to satisfy your desire to ridicule and scorn.

    “One example of perfect human enterprise?” Vioxx is not just some minor imperfection! You excuse this tragedy by your “imperfection” of human enterprise arguments.

    You are saying that the thousands damaged and those killed by Vioxx (that was vetted by your gold standards) is not a major tragedy but a minor imperfection of your application of scientific method. Sick.

    • Skepdude said, on April 14, 2009 at 3:37 PM

      No, I am not saying anything of the sort. You keep imagining words that I am not actually writing. This is no way of having a conversation. You have to address what I say, not what I don’t say.

      I challenged you to give me one example of a human enterprise that is 100% perfect, all the time. Give me one example if you can.

  6. Rightside said, on April 14, 2009 at 4:08 PM

    But that is not the point of this post. It is about “a case study of a bad study”.

    The trials for Vioxx neither predicted the tragic lack of safety nor efficacy, yet they would score high on the scale that determines a good study from a bad one.

    So if Vioxx trials score high, then this whole exercise is completely useless except as a bit of trickery to ridicule a homeopathic study.

    • Skepdude said, on April 14, 2009 at 7:54 PM

      Actually it is exactly the point. There are basic requirements for a study to be considered reliable. If it does not meet these minimum requirements you cannot rely on the conclusion.

      Nevertheless, a good study does not necessarily pick up all the side effects, given the size constraints. There is nothing surprising with the fact that a side effect will not be spotted until a large number of people are using the drug. A study is not going to answer every question. Viagra was not meant to do what it does now, it was only after it was being used widely that its other effect came to light (or so the popular story goes).

      Think of it this way: in order to be picked up by an NBA team you must meet a minimum point scoring requirement, but that does not mean you’ll score every free throw! Same goes for studies. Or think of sharp shooters. In order to be one in the army you have to be able to make certain shots reliably, but that does not mean you’ll never miss!

  7. Rightside said, on April 15, 2009 at 8:42 AM

    Your banal analogies are an insult to the families of the people who died from Vioxx.


    You are exposing a system of drug discovery and application that does not appear to be very scientifically sound or precise to me. It is quite loosey-goosey. So be it. That is the nature of the clinical application of medicine including surgery. But then don’t apply precise, absolute and perfected expectations of homeopathy. In its clinical application it appears to also have gained some momentum, effective results and rectitude without the deaths and destructive side effects.

    • Skepdude said, on April 15, 2009 at 9:04 AM

      There you go pulling the Offense Gambit on me. Are you going down a list of fallacies, making sure you use them all?

      I am trying to have an honest conversation here, but you don’t seem interested in that; and now I’ve lost interest too. You’re hopeless!

  8. Rightside said, on April 15, 2009 at 11:01 AM

    Sorry you have lost a convert to the odd way you apply science.

    I have to use another word which I know you may not like, which is that you sound like a disappointed evangelist.

    It is very hard to get over the offense of you downplaying and excusing with “science”, multiple deaths as an effect.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: