Skepfeeds-The Best Skeptical blogs of the day

Case study : Bad Studies

Posted in Skepdude by Skepdude on April 13, 2009

Homeopathy enthusiasts are pointing to yet another bad study as proof that homeopathy has a basis in science. This one is from March 2007 and is titled “Homeopathic and conventional treatment for acute respiratory and ear complaints: A comparative study on outcome in the primary care setting“. It purports to show that homeopathic remedies worked just as well as conventional remedies to treat “accute runny nose, sore throat, ear pain, sinus pain or cough”. You can read the details at the link above. What I want to concentrate here is on just how badly designed this study is. It seems to me as though it was set up specifically to produce the sort of answer that the homeopaths were after. Nevertheless let me present my arguments and you can make up your own mind.

Before we look at this specific study, let us go over the basic things to look at when reviewing any study. How well the study is designed and how well it conforms to certain rules has a profound effect on how much reliance we can put in its results. One way of assesing how good a study is, is what is known as the Jadad Scale. The Jadad scale is a simple checklist that helps you decide how well designed, therefore how trustworthy a clinical study is. It concentrates on the following areas: Randomizatin, Double Bliding, Withdrawals and Drop Outs. Here is a typical way of calculating the Jadad Score.

Basic Jadad Score is assessed based on the answer to the following 5 questions.

The maximum score is 5.

Question Yes No

1. Was the study described as random? 1 0

2. Was the randomization scheme described and appropriate? 1 0

3. Was the study described as double-blind? 1 0

4. Was the method of double blinding appropriate? (Were both the patient and the assessor appropriately blinded?) 1 0

5. Was there a description of dropouts and withdrawals? 1 0

Quality Assessment Based on Jadad Score

Range of Score Quality

0–2 Low

3–5 High

So let us go over the homeopath’s study and see how it ranks based on the Jadad Score.

1-Was the study described as random? NO – 0 points (cumulative)


The study was designed as an international, multi-centre, comparative cohort study of non-randomised design.

Not only that , but the patients were asked which group they wanted to be in, homeopathy or conventional medicine (misspelling of the word enrollment is theirs not mine! Also emphasis is added by me).

Upon enrolment in the study, patients, or the patients’ legal guardians were asked for their treatment preference. In the homeopathy group, 81% of patients had a preference for homeopathy, 18% had no treatment preference. In the conventional group, 55% of the patients’ preferred conventional treatment, 2% homeopathy and 43% had no treatment preference.

Fun Fact – 81% of the patients in the homeopathy group had chosen homeopathy and the results from the homeopathy group were…drum roll….86.9% reported complete recovery. Can you say placebo?

2-Was the randomization scheme described and appropriate? There was none. – 0 points (cumulative)

Randomization is very important when setting up clinical studies. Not only is it important to randomize the patients, but also how you randomize them matters. Different methods of randomization rank higher than others. According to Wikipedia:

Randomisation is a process to remove potential distortion of statistical results arising from the manner in which the trial is conducted, in particular in the selection of subjects. Studies have indicated, for example, that nonrandomised trials are more likely to show a positive result for a new treatment than for an established conventional one.

I haven’t checked that claim on the last sentence, so take it with a grain of salt, even though it does make sense.

3-Was the study described as double-blind? No, there was no blinding whatsoever, doctors knew what treatement each patient was getting and patients knew it too (they got to choose remember) – 0 points (cumulative)

Since it was not possible to blind patients for their treatment, potential reporting bias from patient’s expectations may have influenced the outcome.

You think?

4-Was the method of double blinding appropriate? (Were both the patient and the assessor appropriately blinded?) There was no double blinding, there wasn’t even single blinding. – 0 points (cumulative)

This is where I would start worrying if I was trying to use this study to prove my point. We’re up to question 4 of 5 and they have 0 points!

5-Was there a description of dropouts and withdrawals? Not even close, they only mention that 6 people who got no treatment were dropped. – 0 points TOTAL!

This study ranks as possibly the worst designed study you could come up with. There was no randomization, no blinding of any sort, let alone double, no control group, in other words nothing that would lend it even a slight amount of legitimacy. The authors seem to have benn aware of this, for they make sure to make the following point: None of that stuff really matters, our study is good enough as it is! Notes in red are my comments.

Objective data collection and evaluation is needed to assist physicians in patient care and advance the quality of medical practice [2] This study will presumably be objective!. Clinical trials, especially randomised controlled trials (RCTs), are generally accepted as producing the highest level of evidence for medical interventions. I feel there’s a “but” coming! Driven by the discovery of new pharmaceutical substances, demands from regulatory authorities for clinical data and the need of physicians for evidence based treatment strategies, the methodology of RCTs became the subject of research itself. Within this context, the strengths and weaknesses of such trials have been debated [3]. Placebo-controlled RCTs are indispensable for the development of pharmaceutical agents with unknown efficacy and safety profiles Such as maybe homeopathic agents. On the other hand if the efficacy and safety of an agent is known why would one even bother to do a study?. Their limitations result from highly standardized study protocols and patient populations, which may create artificial situations that differ from daily practice Oh, I see they are more tightly controlled and have stricter requirements, and THAT makes them problematic. What? . Moreover, even the fact that patients are enrolled into a placebo-controlled clinical trial will influence treatment outcome, sometimes leading to high placebo or low verum response rates [4] Somehow I did not think it was a matter of high or low, I thought it was a matter of the truest measure which is the point of the control groups. Further, proper blinding should guarantee the truest results possible. Consequently, more practice-based studies have been developed such as pragmatic RCT’s or non-randomised cohort studies. In other words, when you can’t live up to these standards make up more lax standards and claim they are just as good. Pathetic! Especially non-interventional outcomes studies have only few inclusion and exclusion criteria. Therefore they may provide information about a broad and heterogenous patient population thus resulting in high external validity for daily medical practice Actually lack of controls will result in exactly the opposite, it will be useless for daily medical practice. It may provide a good gauge for people’s ability to deceive themselves though. However, the fact that patients are not randomly assigned to treatments in such outcome studies may lead to baseline differences between groups and makes the interpretation of the results more susceptible to bias. May? That’s putting it mildly! This disadvantage may be overcome, at least in part, by the application of statistical methods to control for baseline differences between treatment groups No it can’t, otherwise randomization would not be required, EVER. Good statistics can never make up for bad data. Statistics rely on the data itsel. The above claim makes no sense!

Fun Fact –

Apart from the ongoing discussion about clinical evidence, complementary therapies are well integrated into primary care in most Western countries

Yeah appart from the fact that CAM has not been shown to work, IT IS POPULAR. Good enough for me!


This study is horrendously designed. It lacks all of the basic requirements that every clinical trial should have, such as randomization, double blinding, control group etc.  Based on that fact alone, regardless of the sample size, regardless of how careful and precise the statistics, the results of such study will be completely unreliable. The data set is corrupted due to the lack of controls, as such it does not matter how carefully you analyze it, the result would be meaningless. Even if it had told us that homeopathy is useless, we would still have to ignore it. And ignore it, I, we and all the science based community will. Sorry homeopaths, you’re still stuck at 0. Good luck next time.

Homeopaths: fallacies, lies, and dangerous advice

Posted in Journey Through a Burning Mind by Skepdude on April 13, 2009

Since this is the World Homeopathy Awareness Week, I might be spending more time on this particular well of credulity. I will also re-post my piece from last year’s WHAW. But today, I’ll be talking about Melanie Grimes, a homeopath who writes for HealthNews as a health “expert”. The irony though is painful -you will soon see that if “expert” was to be used in the same sentence as Melanie, then that would be: “Melanie Grimes is the exact opposite of a health expert“. So let’s start the fun, shall we?

I had a look at 3-4 of her articles*, and I can assure you they are filled with fallacious arguments of the worst kind; a very bad understanding of modern scientific research; misrepresentation or outright ignorance of the relevant scientific literature; and propagation of very dangerous homeopathic beliefs as to what their pet therapy can treat (from cancer to diabetes, it’s all there).

Starting with her piece on this year’s WHAW (starting slowly with some common stuff):

Homeopathy provides an effective and gently way to treat allergies. Using potentized medicines, homeopaths prescribe minute doses to treat both the acute reaction to allergens, as well as the cause.

I hope you have spotted that subtle piece of misinformation: “minute doses”. It’s not minute doses actually. It’s non-existent doses usually. The most common potencies of homeopathic remedies are 12C and above -a dilution so high that no molecule from the original substance remains in the remedy!

But the most interesting claim is that homeopathy is effective for allergies. In fact, this is a very common claim of homeopaths but is there any evidence to back it up? Readers of this blog already know the answer: no.

A quick search in PubMed brings up some relevant reviews [1][2][3], none of them recommending homeopathy (or CAM in general) for diagnosing or treating allergies. Quoting from “Systematic review of complementary and alternative medicine for rhinitis and asthma”[1]:

Some positive results were described with homeopathy in good-quality trials in rhinitis, but a number of negative studies were also found. Therefore it is not possible to provide evidence-based recommendations for homeopathy in the treatment of allergic rhinitis, and further trials are needed. A limited number of studies of herbal remedies showed some efficacy in rhinitis and asthma, but the studies were too few to make recommendations. There are also unresolved safety concerns. Therapeutic efficacy of complementary-alternative treatments for rhinitis and asthma is not supported by currently available evidence. [emphasis mine]


Important Studied Page Update

Posted in Skepdude by Skepdude on April 13, 2009

Hi there. As you know I am keeping an “Important Studies” page where I will link to studies on which I base my various stances on different Alternative Medicine Modalities. I have just updated it by adding a second study/analysis to the Homeopathy section and adding a new section and a new study in the General/Multiple CAMs studies. Check them out and send me links to other studies as comments on that page. Thank you.