Here's an example of how osteopathic research should be performed (with sham control group and large sample size):
http://www.om-pc.com/content/4/1/2
That study looks pretty compelling...
To the MD folks in the thread what's wrong with it?
Trying to get this thread headed in a better direction.
This is a really interesting study. It was designed as a randomized, double blind, with both sham (positive) and negative controls.
As has been mentioned above, designing a positive control (i.e. "Sham OMT") is complicated. You need to find some way to "do something" to patients that's similar enough to OMT that the patients can't tell the difference (i.e. it's as similar to OMT as it can be, without doing the "core" of OMT).
So, overall, the study is reasonably well designed and implemented. But it's not perfect.
One big problem is the blinding. At the end of the study they ask the patients whether they thought they were in the sham or active treatment groups. The majority were able to tell. This is a huge problem, and is a failure of blinding. Part of the problem could be too much of a difference between OMT and the sham (so that patients can tell), or perhaps that the OMT providers were talking to the patients while being treated, and gave away what they were doing. In any case, it creates problems. Since patients knew they were getting OMT (despite the attempted blinding), they are likely to report improvements with OMT -- they want to make the investigators happy. Avoiding this is the whole point of blinding. Plus, it's possible that the physicians caring for the patients equally had a "leak" of blinding (i.e. they shouldn't be able to witness an OMT provider going in to treat their patient -- the patients should be moved to a different room / area and then have their treatments, or it's possible that the patients who knew their treatments told their providers). In that case, providers might try to discharge those patients a bit faster.
But, the study has a much, much bigger problem. The Intention-to-treat analysis (ITT) showed no effect. The Per Protocol (PP) analysis showed a benefit. To review, an ITT analysis treats everyone randomized to OMT (even those who withdraw or ultimately don't get OMT) in the OMT group. A per-protocol analysis takes people randomized to OMT but who didn't get it and either removes them from the study, or adds them to the control group.
The authors suggest that the PP analysis shows a benefit because those people actually got OMT, and that the ITT analysis doesn't because the OMT effect is "washed out" by people randomized to OMT who ultimately didn't receive it. And that sounds reasonable, and it's theoretically true.
But, it's completely unacceptable.
Why? Because there are two other reasons why a PP analysis can show a benefit when an ITT doesn't. First, if your intervention (OMT) actually causes harm, that can cause people to drop out of the study. For example, consider a drug for cancer that causes severe side effects (perhaps kidney failure requiring HD) in 20% of cases that results in stopping the drug, but helps the remaining 80%. If you just "dropped" those 20% of people from the analysis, it would look like the drug works in everyone and is safe. Now, I agree that this is unlikely for OMT, as it seems unlikely that it could cause harm in some way.
The other reason is "sick dropout". Let's assume (for the sake of argument) that OMT does not work. You randomiaze 100 patients to OMT, and another 100 get a placebo treatment. Of the 100 patients randomized, 20 get really sick -- too sick for OMT and so it is stopped. Now, you have 80 patients left. If you compare how those 80 patients do with the 100 placebo patients, you're likely to see a "benefit". That's because, on average, 20 of those placebo patients are going to get "really sick" also but you can't drop out of the placebo arm (since there really isn't anything to stop). And, it's even worse if you move the dropouts to the placebo arm.
Bottom line -- if the ITT analysis is negative, you need to stop. Looking at a PP analysis is not appropriate, and is "data mining" looking for the result you want, not the study's actual result. So, this study shows that the intervention studied was no different than placebo.
Before I get pounced on, that does not mean that all OMT doesn't work (although that's a possible explanation).
Earlier on the thread someone was "poo-poo'ing" the placebo effect and mentioned that it couldn't be very significant. That's clearly not true. The RCT's looking at SSRI's showed a placebo effect of at least 20-30% (i.e. 20-30% of depressed patients given a sugar pill felt better). The placebo effect of a drug on cancer progression is often very small, but on subjective feelings is often quite strong.
