Does it really matter who gives assessments? (Rel. Research & Scope)

This forum made possible through the generous support of SDN members, donors, and sponsors. Thank you.

Therapist4Chnge

Neuropsych Ninja
Moderator Emeritus
15+ Year Member
Joined
Oct 7, 2006
Messages
22,365
Reaction score
4,256
I was catching up on my blog reading when I came across a post at Clinical Psychology & Psychiatry: A Closer Look, which looks at a particularly poor study that found its way into last month's Archives of General Psychiatry. It is scary that it is so commonplace to have such poorly designed studies make it into peer-reviewed journals. Below is the post, with my bolding of the worrisome parts.

As an aside, I'd also plug "Furious Seasons", "The Last Psychiatrist" (SDN member), and "Pharmalot". PandaBear had a nice blog too, but he stopped last year.

---------------------------------------------------------------------
A study in the December 2008 issue of the Archives of General Psychiatry concluded that almost half of college aged Americans suffered from a DSM-IV disorder over a one-year timeframe. Yes, I am behind the curve on this one -- Furious Seasons was all over this last month (1, 2). Rather than rant about the very odd idea that half of young adults are suffering from a mental disorder, I want to start by mentioning one aspect of the study -- perhaps the most important one. Let's look at how the diagnoses were assigned. To quote from the study:
All of the diagnoses were made according to DSM-IV criteria using the National Institute on Alcohol Abuse and Alcoholism Alcohol Use Disorder and Associated Disabilities Interview Schedule–DSM-IV version, a valid and reliable fully structured diagnostic interview designed for use by professional interviewers who are not clinicians.
If the interviewers are not clinicians, on what basis are they trained to understand what makes for truly significant distress that might justify a mental health diagnosis versus someone who is suffering from more mild symptoms that do not comprise a mental disorder? Here's some information from a different study that used a different slice of the same overall dataset on which the December 2008 study was based:
Approximately 1800 lay interviewers from the US Bureau of the Census administered the NESARC using laptop computer–assisted software that included built-in skip, logic, and consistency checks. On average, the interviewers had 5 years' experience working on census and other health-related national surveys. The interviewers completed 10 days of training. This was standardized through centralized training sessions under the direction of NIAAA and census headquarters staff.
So the figures that will be trotted out in the media ad infinitum about the shoddy mental health of American youth are based on laptop-assisted interviews conducted by people who apparently have no formal training in mental health. Maybe mental health and related disability are really so easy to assess that we don't need experienced, formally trained interviewers. If that's the case, maybe we should just have Census Bureau interviewers provide initial mental health assessments in clinical care settings -- after all, if they are such good mental disorder detectors, couldn't we just train a bunch of interviewers rather than spend millions of dollars training and paying mental health professionals? Think of the savings!


I mean no disrespect toward the Census Bureau interviewers. They are performing important work that in many instances helps us to better understand the health of the nation. All I'm saying is that we might want to avoid uncritically accepting judgments of our nation's mental health based on interviewers who lack mental health training and experience.
---------------------------------------------------------------------


I couldn't agree more with the writer in regard to the importance of properly trained professionals doing assessment and diagnosing of people, and not leave it up to screeners and non-mental health professionals.

It saddens me that something like this can get published, and furthermore that people accept it because it was published. I know many studies don't have MS/Ph.D/Psy.D. people involved in the research, but interviewing and evaluating symptoms isn't something I'd trust to someone off the street with no real training. Oh wait...they received 10 days of training. :rolleyes:

Thoughts?



Members don't see this ad.
 
Neurologists, as an example, give neuropsych tests with no training (you know, because physicians are automatically competent at everything). At neurology conferences (e.g., American Academic of Neurology), I have seen many computer programs designed to give neuropsychological tests and interpretive scores. . . It's very frustrating.

I've seen this more and more....which is one reason why I posted this thread. It started with "screeners"....but now it seems like everyone is giving assessments in the medical setting, and most have no idea what they are doing.
 
Yes it matters who does assessments. It is the purview of clinical psychology and not other professions. I wonder if this issues reflects the general reduction in professionalization we see in mental health. Psychological tests should only be given and interpreted by psychologists. Don't the licensure laws in the states protect our ability to maintain control over these assessment tools to prevent misuse. I have seen MD psychiatrists give the MMPI-2 despite their complete lack of training to be qualified to do so. They would seem to be practicing psychology without a license. I have never been able to understand that. Or does the medical license legally cover any and all procedures they chose to give??
 
Members don't see this ad :)
I have somewhat mixed feelings on the study.

On the one hand, I agree that its probably not nearly as accurate as it would be if the testing was done by experienced clinicians. We need to watch ourselves since we don't want this to become a pervasive problem in our profession.

On the other hand...does that make it a bad study? I'm not willing to go that far. Perhaps bad writing if they don't discuss the consequences of having untrained individuals diagnosing without discussions of reliability, etc. To be honest, I'm not familiar enough with the measure to have a feel for how trustworthy it is. The question I have is how else would we get this information. Will you shell out the money for doctoral level folks to conduct 40,000 interviews? What if the interviews were given by these folks, but reviewed by doctoral level folks? What IS an acceptable level of training for interviewers in studies like this? Undergrad RAs? 1st year grad students? 5th year grad students? Only psychologists with >20 years experience? Its not an easy question, I certainly don't have an answer.

I see the issue as less about study design, and more about publication bias and media abuse. Medical journals, even more than psychology journals, often want write-ups so brief it CAN'T be anything but irresponsible science. There's only so much you can discuss when you only have a few hundred words. Not to mention if you spend too much time discussing the faults you're basically giving them an excuse to reject you. Of course, the media doesn't care at ALL about accuracy. They're looking to draw headlines, and it doesn't matter how accurate something is...heck if they need to distort it to get those headlines, that's fine too.

The data is what it is. Personally, I think its valuable information to have, as long as its placed in the appropriate context. The issue I have is that the system is set up such that its bad for your career to do that.

As for testing in clinical settings...this is a different matter. I assume people with high levels of symptoms were referred to actual clinicians for further evaluation in the archives study. Its very different if medical decisions are actually made off faulty information because its "cheaper" then good information. So many factors contribute to this...though managed care is probably #1 in my eyes.
 
Neurologists, as an example, give neuropsych tests with no training

I became aware of this practice after witnessing the neurology nurse dipping into our psychometrist's supply closet for materials. I assumed they were doing some kind of screening, although I'm not certain how they are using (or misusing) that information.

On occasion we get a doc that will try to request some specific tests for say a DBS eval. No offense, but please leave the heavy pencil-and-paper lifting to us!
 
i blame the test publishers. MDs can order anything they want. everyone else has to prove some education in test admin and stats. weird.
 
Unfortunately the testing companies are in favor of other professionals using their products (more $$). The test companies give afternoon/weekend seminars to "train" people on the instruments.

The APA needs to step up and do something about this...but as usual, they fail to make a stand about anything.
 
tests and measures are the domain of psychologists indeed. we should protect our turf and do it generally unabashedly. its in the patient's best interest.

however, if only experienced people administered tests, i'd never get to administer my first. so, i can see that the process needs a way in for psychologists.

i guess it's the degradation of information down the line that can only be improved through (more than a few sessions of) training and experience which concerns me.
 
Top