Determining If A Study Is Reliable.

This forum made possible through the generous support of SDN members, donors, and sponsors. Thank you.

MaddieMay

Full Member
10+ Year Member
15+ Year Member
Joined
Feb 25, 2008
Messages
599
Reaction score
2
Do you think that a study must fit a certain criteria before it is deemed reliable, or do you believe studies fall on a spectrum of reliability depending on how well they fulfill the reliability criteria?

Members don't see this ad.
 
First off, are you referring to statistical reliability, or reliability in a more general sense? Just want to make sure to clarify since statistical reliability has somewhat clearer standards. I'll assume you are referring to a more general "Can I trust what the study says" definition.

I actually sort of use both. I have a threshold below which I will deem it pure and utter garbage. Things that clearly use flawed methodology (e.g. manipulation that doesn't actually do what think think it does, that one study I read a few years back where someone used the BDI to measure the efficacy of their mood induction, etc.), try to draw conclusions that need a control group without actually having one, things that CLEARLY don't have a representative sample (you would not BELIEVE how much research is out there that was run on say, treatment-seeking individuals at clinic x, that authors seem to think generalizes to all populations.) Nothing wrong with running studies on treatment-seekers in clinics, just don't pretend its reflective of everyone;) But I digress.

Once something has passed the "crap" threshold, that doesn't mean I trust it absolutely. I typically lump it into a "Needs further investigation" category, and if it seems like alot of studies are finding similar results, than I may trust those findings. After that there is a spectrum of reliability based off way too many factors to list, many of which are "soft" factors to begin with that may or may not relate or be necessary in a given study.

So really its both. I think a study needs to meet certain criteria (that vary based on the kind of research) for me to deem it anything other than a waste of time. After that, there's still a spectrum of how trustworthy something is.
 
Well, we don't really measure rigor of a study by "reliability" alone. Reliability refers to the reproducibility of a result. But obviously, to make for a valid and methodically sound study, one one needs reliable instruments for measuring the construct or phenomena. If as studys' results/findings are replicated using the same methodology and population by future studies, one may be able to say that the original study has "reliable" findings. However, this doesn't necessarily equate to validity. You could be studying something else entirely (not the construct or phenomina you thought), and since other studies are replicating that methodology, you just get the same flawed results over and over.

Reliability of instruments comes in many forms, split-half.....-internal consistency, etc. The reliability of an instrument for clinical purposes should be around the .90 range (Chronbachs alpha), for research, .80 or above.

PS: Ollie got it, and I was approaching mine mainly from a statistical perspective. :)
 
Members don't see this ad :)
Yes, thank you both for clarifying. I am sleep deprived from last night's crazy thunderstorms.

I was thinking in terms of validity and "Can I trust what this study says?" but all of the information you both mentioned was very helpful.

I have foolishly gotten into a debate about whether marijuana administered through an inhaler/nebulizer, ingested or via suppository is still detrimental to your health in terms of PD and depression after discontinuation of heavy use, impaired ability to drive, memory problems, etc. And first I'm trying to explain why essays from doobie.com do not meet the criteria for a valid study. What a fool's errand! Time for a nap! :)
 
....:laugh: Well ok then. Check out this link from Dr. Park's lab at Vandy. http://www.psy.vanderbilt.edu/faculty/sohee/
Go to publications. Many can be downloaded for free on there. She does some work with cannabis use and susceptibility to depression and psychosis.

PS: Might also want to mention to this person that, with a name like doobie.com, there is likely to be a slight selection bias of studies they chose to post......:) I doubt doobie.com has done an exhaustive literature search on the topic if you know what I mean....lol
 
Most excellent link! Thank you.

P.S. I actually made up the name doobie.com, but don't tell anyone. :)
 
For a second I was wondering how I went through life not knowing about "doobie.com" until this point. The domain name is registered, the site is just blank. Such wasted opportunity :D
 
Top