Test selection issues

This forum made possible through the generous support of SDN members, donors, and sponsors. Thank you.

WisNeuro

Board Certified in Clinical Neuropsychology
15+ Year Member
Joined
Feb 15, 2009
Messages
19,220
Reaction score
26,769
I wish this test would die a fiery death. The only positive research you will see is by people who have a financial interest in the test. On the other side you have a mountain of research that deals with the many problems (lack of construct validity, lack of test-retest, inconsistent FA, reliance on the Forer effect, etc etc). A parlor game at best and should be left behind as a defunct psychological concept with the Rorschach.
 
I wish this test would die a fiery death. A parlor game at best and should be left behind as a defunct psychological concept with the Rorschach.

While I agree with you on the MBTI, you should maybe talk to your supervisor about your angry and morbid desires 😛


Also, they have updated the Exner model for the Rorschach. It is now called R-PAS. Its validity and reliability are much higher and it has updated norms and a new sample. Here are some articles:

http://www.r-pas.org/articles.aspx

You may be interested in this one:

Porcelli, P., Giromini, L., Parolin, L., Pineda, J. A., & Viglione, D. J. (2013). Mirroring activity in the brain and movement determinant in the Rorschach test. Journal of Personality Assessment, 95, 444-456. DOI:10.1080/00223891.2013.775136
 

Attachments

🙂 I prefer to label the anger as "drive." I've seen the R-PAS stuff, but it still hasn't addressed some of the major flaws from the literature I posted in the last discussion of the Rorschach. I still don't see the utility of it, especially based on the time of administration and scoring. I can get far more information in far less time with interview and other measures.
 
🙂 I prefer to label the anger as "drive." I've seen the R-PAS stuff, but it still hasn't addressed some of the major flaws from the literature I posted in the last discussion of the Rorschach. I still don't see the utility of it, especially based on the time of administration and scoring. I can get far more information in far less time with interview and other measures.

In terms of how long it takes to score, I wish this test would die a fiery death.

On the other side of the coin, while it has some flaws (as all tests do), it has some surprising usefulness in certain situations.

psych84, just copy and paste this entire thread into your References section, that will totally work out fine for you.
 
On the other side of the coin, while it has some flaws (as all tests do), it has some surprising usefulness in certain situations.

It may have some usefulness, but what incremental usefulness does it have? Does it do anything unique? Or does it do anything better than or in less time than other measures?
 
It may have some usefulness, but what incremental usefulness does it have? Does it do anything unique? Or does it do anything better than or in less time than other measures?

Within many tests, especially projectives, we all have personal preferences. Let me clarify that I don't love the Rorschach that much, and I find it more intriguing than anything else. Anyways, with my limited experience, I have found the Rorschach useful in finding subtle delusions (and maybe psychosis) that don't present themselves in a clinically obvious way. To specify, while keeping it short, I had a patient who presented with moderate anxiety and a general lack of motivation. Their MMPI had mild elevations with nothing too severe. FSIQ was ~120. Nothing amazing in their intake. This is a unscientific and broad statement: Something just didn't seem right, but I couldn't put my finger on it so I gave them a Rorshach. Anyways, their responses to the Rorschach showed a great deal of morbidity and general despair. Their responses led me to question them more and I figured out they had some esoteric, but significant, delusions.

Here is where I imagine you saying (in a more polite manner), "The TAT would have been easier and faster" "The XXX test would have been better" "A more experienced clinician would have been able to figure this out without a test" ect.

Yes, that is all true. However, I think the Rorschach's ambiguity is somewhat unique; that ambiguity can open some doors. Regardless, long-term, I would probably use the Rorschach maybe twice a year - its just nice to have in the arsenal. Again, personal preference.
 
Agreed with you Cara, the TAT has even worse psychometrics. Psych, I know you can point to this one instance where you may have gotten some insight into a delusion, that may have not gotten picked up in a clinical interview. What's the sensitivity and specificity of this ability of the measure? Anecdotes are good and done, but their plural is not data. You're talking about diagnosing a low base rate disorder (<1%) with a historically unreliable instrument. The false positives are through the roof!
 
I won't bother with the new statistical data backing the R-PAS system, mostly because i'm tired. However, most projectives, in my eyes, are there to give the clinician a medium in which to pull out a patient's un/subconscious thoughts and skewed (yet not always delusional) beliefs or invalid schemas, or whatever CBT people are calling them these days. It's a way to find out more about the patient in a somewhat structured manner.

Here is what I say we do, after months of studying and right before we take the EPPP, we anxiously vomit onto a card, fold it in half and let it dry. Then we use these new psychologically-rich blots as the new "Roschach" system. No need to validate this measure because our vomit will intrinsically be reliable - assuming we passed the EPPP.
 
I won't bother with the new statistical data backing the R-PAS system, mostly because i'm tired. However, most projectives, in my eyes, are there to give the clinician a medium in which to pull out a patient's un/subconscious thoughts and skewed (yet not always delusional) beliefs or invalid schemas, or whatever CBT people are calling them these days. It's a way to find out more about the patient in a somewhat structured manner.

How does this information inform/guide treatment? Is there research to support it's clinical efficacy in this way?

I'm not trying to be a jerk, I just want to see the data that these things reliably do what they claim, and that they actually add something to the patient's outcome? As of now, all I'm seeing is people saying that the Rorschach is an "art" and that they don't need this data, that they just "know" that it helps. We have mounds of actual data that show the fallacy of that attitude.
 
I won't bother with the new statistical data backing the R-PAS system, mostly because i'm tired. However, most projectives, in my eyes, are there to give the clinician a medium in which to pull out a patient's un/subconscious thoughts and skewed (yet not always delusional) beliefs or invalid schemas, or whatever CBT people are calling them these days.

Thats fanatastic. Does it help the patient?
 
Wis, I don't disagree with you, I am mostly playing devil's advocate. I really don't know if the outcome data you're asking about even exists, but look here: http://www.r-pas.org/Articles.aspx

Though, remember, you can't objectively describe "art."

erg, it might be silly for me, but I assume if a clinician knows more about their patient's thoughts and beliefs, it might increase the therapist's ability to address the patient's issues. My own passive-aggressive comments aside, if we threw raw data from most assessments at the patient, they won't know what it means, particularly projectives. It is our job to interpret the tests (which are mediums) and provide the information to the patient, where appropriate.
 
I have a supervisor that believes I was poorly trained in the Rorschach because I was learning the Exner system and thus lost the "art" of the test.

It still interesting how dividing the topic remains. From my read of the literature, the test is valid and reliable for certain diagnoses and useful for some minor situations. However, on the flip side it provides almost no incremental validity compared to other tools with greater psychometric properties.

FWIW, I would only use the Rorschach when I need a low face valid assessment tool because my other personality assessments were invalid and I need corroboration of severe pathology (not depression/anxiety)
 
I don't doubt that people have done studies using the Rorschach. Point me to one of those studies that shows incremental validity of the thing over and above other more valid and reliable measures that take a fraction of the time to administer. I have not seen any good data that says that objective patient outcomes have been improved through the use of the instrument. If those don't exist, how can you justify the time and expense of the test?
 
how can you justify the time and expense of the test?

The time spent scoring definitely cannot be justified; however, I know experts who can abstractly score the Rorschach, but that's the minority.

Cost is nearly free if you print the images from google and photocopied the manual (sarcasm, don't do this). I think online protocols cost $5.
 
The time spent scoring definitely cannot be justified; however, I know experts who can abstractly score the Rorschach, but that's the minority.

Cost is nearly free if you print the images from google and photocopied the manual (sarcasm, don't do this). I think online protocols cost $5.

I meant more the time and expense of using this within a clinical setting. The patient is paying for that time. If the thing doesn't actually do anything, I would make the argument that it's unethical. You're selling them some kind of placebo. But, instead of making them fell better, it just gives the clinician some false sense that they just added something profound to the mix.
 
I meant more the time and expense of using this within a clinical setting. The patient is paying for that time. If the thing doesn't actually do anything, I would make the argument that it's unethical. You're selling them some kind of placebo. But, instead of making them fell better, it just gives the clinician some false sense that they just added something profound to the mix.
I think the Rorshach, used in the R-PAS system, can be useful in certain situations, especially where thought disorder is suspected but not otherwise evident. It also can give you a useful observable behavioral problem solving sample that can be interesting corroborating data when integrated with other test findings. That said, I find it interesting that will all the controversy, being able to adminster, score and provide feedback using projectives remains an "essential" element of testing competency by typical APA site visitors.
 
I disagree with the thought disorder sentiment exactly. The old CS system was taken to task for it's overdiagnosis of thought disorder, to the effect of 1 in 6 in control populations. How did they RPAS solve this? And, has it been demonstrated in independent labs?

Also, I ask again, what incremental validity does it give over other instruments? And, how does this change either diagnosis or patient outcomes?
 
I disagree with the thought disorder sentiment exactly. The old CS system was taken to task for it's overdiagnosis of thought disorder, to the effect of 1 in 6 in control populations. How did they RPAS solve this? And, has it been demonstrated in independent labs?

Wis, you make a good point and ask good questions. I have no idea, but I am asking one of the R-PAS researchers and I will get back to you - assuming he responds within a reasonable time.
 
I find that hard to believe, given that many rigorous clinical science programs do not teach projectives - it's a lot of wasted training time for something of little clinical utility.

I also had little-to-no required training in projectives while in grad school. More could've been available at off-site externships if I'd desired, but it wasn't pushed particularly hard in my program.
 
We were trained in projectives in grad school, although we were also introduced to their criticisms. I'm a strong believer that you should be well aware of the strengths and limitations of the instruments you use in assessment. And, if the data is lacking, it shouldn't be used. As I've said before, we are a science, not an art.
 
What internship programs outside northeast view the ror as "essential?" My experience in the heartland is much, much different. The complete opposite, actually. I have never seen it given outside of a pp setting...where that time can be justified.
 
What internship programs outside northeast view the ror as "essential?" My experience in the heartland is much, much different. The complete opposite, actually. I have never seen it given outside of a pp setting...where that time can be justified.

Agreed, it may vary regionally, but my experience in the Midwest and the Southwest here has been that as well. I have yet to see a Rorschach in an assessment report.
 
So since the MMPI-2 is thought to be the most widely used instrument for clinical assesment, what would be the 2nd most used?
 
Chicago is still a stronghold for psychodynamic, psychoanalytic, and Rorschach activities.
 
So since the MMPI-2 is thought to be the most widely used instrument for clinical assesment, what would be the 2nd most used?

Good question. I'd imagine the WAIS has to be near the top given its utility in a variety of contexts. But for psychopathology/personality rating scales and the like, I'm not sure...runner-up to the MMPI-2 would possibly be the PAI (as mentioned) or MCMI. This is of course excluding some of the brief self-report measures like the BDI-II and BAI.
 
Good question. I'd imagine the WAIS has to be near the top given its utility in a variety of contexts. But for psychopathology/personality rating scales and the like, I'm not sure...runner-up to the MMPI-2 would possibly be the PAI (as mentioned) or MCMI. This is of course excluding some of the brief self-report measures like the BDI-II and BAI.

I guess the Psychiatrist/Psychologist would have to use clinical judgment to decide if using such assesments is neccesary? Is there any research that suggests that these tests are over-used or under-used, (on what percent of patients), and if there is any attempt to save money by not administering the test?
 
There are few clinical cases where it is necessary, per se. The question is: is the information needed/will it translate to any substantial changes in treatment planning?
 
I attend grad school in the Midwest and the supervisor at one of my practicum sites, which was all assessment, absolutely loved it.
 
I know, I'm just adding to the geographical discussion.
 
erg, it might be silly for me, but I assume if a clinician knows more about their patient's thoughts and beliefs, it might increase the therapist's ability to address the patient's issues. My own passive-aggressive comments aside, if we threw raw data from most assessments at the patient, they won't know what it means, particularly projectives. It is our job to interpret the tests (which are mediums) and provide the information to the patient, where appropriate.

And to finally address this, I will state what I have stated many times in the past.

Of course one should get to know their patients and yes, of course more info can be useful. But it isnt necessarily useful...especially within short term treatment contexts. I think there are exceedingly few instances where this vague notion of a "rick, deep, and comprehensive understanding of personality function" is needed, or even useful, frankly. Sorry grad students, but its true.

Forensic cases, in which case a measure where the sensitivity and specificity is unknown (supposedly) would not be appopriate, and preparing for a couple years on the couch. Thats about all that I can think of. Not that we shouldn't try to understand our patients and continue to gather info throughout treatment, but I find the above notion to be shaky (on empirical grounds) and not a particularly necessary ingredient for sucessful treatment in most cases. It's certainly not a mindset that psychologist can or should bring to intergrated primary care/mental healthcare system. Its just not pragmatic.
 
Last edited:
That said, I find it interesting that will all the controversy, being able to adminster, score and provide feedback using projectives remains an "essential" element of testing competency by typical APA site visitors.

Depends heavily on the program. I think the site supervisors are usually drawn from the same program model so clinical scientist places get clinical scientist site visitors (none of whom I can imagine requiring substantive projective experience), etc. My Rorschach training consisted of a single class framed as a debate on whether or not data supported its use in clinical settings (landslide "no"). This was admittedly before R-PAS came out and I haven't had a chance to dig into the data on that yet to form an opinion on whether it is a substantive enough improvement to justify its use. My grad program is in the southeast and I'm now going on internship in the northeast (and had many interviews there) where no one cared a lick that I'd never used it. These were AMCs/VAs though.

From what I've seen, it seems popular in a few large cities (NYC, Boston, Chicago) and forensic settings (likely for the need for low face validity per comments above...though arguably that is a situation where its questionable psychometric properties are VERY concerning). I've honestly thought about getting training in it just out of interest in the debate, but given the infinite volume of information to learn in this field, it has been very far down the priority list. When given a choice between projectives or new stats techniques, there has never really been a question about what it would be more beneficial for my career to learn so I've had to put it on the back burner - at least until I see some data convincing me it is worth my time!
 
Not that we shouldn't try to understand our patients and continue to gather info throughout treatment, but I find the above notion to be shaky (on empirical grounds) and not a particularly necessary ingredient for sucessful treatment in most cases. It's certainly not a mindset that psychologist can or should bring to intergrated primary care/mental healthcare system. Its just not pragmatic.

Fully agree - there seems to be significant opposition to "efficiency" in the field that I never fully understood. If I could wave a magic therapy wand and make people completely better in 10 minutes I think that would be fantastic...but I get the impression a substantial portion of folks in the field would be horrified. Obviously its important to make sure quality of care is equivalent, but if you are planning on charging people it doesn't seem ethical to always err on the side of "more treatment/assessment/etc. is better". If it improves outcomes - great, do it. A disturbingly large portion of activities in the field seem to be more for our benefit than the patients though.
 
There are few clinical cases where it is necessary, per se. The question is: is the information needed/will it translate to any substantial changes in treatment planning?

This…if you are referencing the Rorschach. I personally do not believe in it nor support the teaching of it. I too went through training before the RPAS push came about, so having to learn the Extner System was painful and almost a complete waste of time (outside of the time I spent reading critique articles). The only places I saw it as a training requirement for internship/fellowship was in the NYC area. Some (mostly old guard) clinicians swear by it, but I don't see how it'd be actually clinically indicated and relevant for 99% of cases.

--

As for the MMPI/personality assessment…other common options I have seen (in order of frequency):
1. MCMI
2. PAI
3. CPI (California Personality Inventory)

In a hospital/rehab setting I prefer to use the BHI-2 (Battery for Health Improvement-2). It isn't a pure personality measure, but it is pretty useful for TBI/Stroke/etc. I use it in conjunction with my typical neuropsych battery, though it really helps bring some data to compare against prior documentation, clinical interview, outside provider feedback, etc.
 
Last edited:
Edited
 
Last edited:
Of course one should get to know their patients and yes, of course more info can be useful. But it isnt necessarily useful...especially within short term treatment contexts. I think there are exceedingly few instances where this vague notion of a "rick, deep, and comprehensive understanding of personality function" is needed, or even useful, frankly. Sorry grad students, but its true.]

Forensic cases, in which case a measure where the sensitivity and specificity is unknown (supposedly) would not be appopriate, and preparing for a couple years on the couch. Thats about all that I can think of. Not that we shouldn't try to understand our patients and continue to gather info throughout treatment, but I find the above notion to be shaky (on empirical grounds) and not a particularly necessary ingredient for sucessful treatment in most cases. It's certainly not a mindset that psychologist can or should bring to intergrated primary care/mental healthcare system. Its just not pragmatic.

Not to change the subject of this thread, but that same argument you made there (in bold), is essentially the argument the anti custody-evaluation psychologist used in her arguments.

Her argument was essentially:
1. People that are largley able parents and generally normal people are being evaluated in a way that is almost always comprehensive in nature, leading to lots of billable hours for the Psychologist. People seem to end up in these custody battles largely due to a fear of the justice system and the idea that they will lose their child if they don't fight for the child in this manner, not because they aren't able to take care of their kids, or because they have some serious psychopathology.
2.Most of the tests used in a custody evaluation have no relevance in terms of parenting capacity. IQ tests are fairly standard but what is the relevance of IQ in terms of custody evaluation? Where is the research that shows that parents with higher IQ's make better parents? There is none. If the IQ test is simply used to rule out ******ation, does it really take a Psychologist to suspect such a thing? Couldn't an average person with literally no training suspect such a thing?

What I gathered from your comment erg was that more comprehensive testing is not neccesarily useful because the more information we have, the more inferences/judgements we have to make, and that often leads to worst decisions that are not evidence-driven.
 
If the IQ test is simply used to rule out ******ation, does it really take a Psychologist to suspect such a thing? Couldn't an average person with literally no training suspect such a thing?

I want data, not pure clinical judgment. Meehl has published a number of articles on the limits of clinical judgement. Unfortunately many clinicians (-ologists, -iatrists, and others) still choose to believe that "they just know" or "in their expert opinion (based solely on a clinical interview)" they can accurately assess someone and don't need objective measures/data.
 
What I gathered from your comment erg was that more comprehensive testing is not neccesarily useful because the more information we have, the more inferences/judgements we have to make, and that often leads to worst decisions that are not evidence-driven.

Actually, I don't believe that this was what erg was getting at. I think he was implying that I don't need to know what psychosexual stage someone is stuck in, or how they felt about their potty training, to treat their Panic disorder or PTSD from a recent traumatic event. While some of teh information may be interesting to the clinician, it does not lead to improved clinical outcomes.
 
What I gathered from your comment erg was that more comprehensive testing is not neccesarily useful because the more information we have, the more inferences/judgements we have to make, and that often leads to worst decisions that are not evidence-driven.

Um, yea..no. Not sure how you got that. The more data points we have, the better (more reliable) our predictions can be, assuming we use the data correctly. Multivariate statistics 101: Variance increases reliablity.

Do I need all that info to treat your depression? In most cases, no, not at all.
 
Not to change the subject of this thread, but that same argument you made there (in bold), is essentially the argument the anti custody-evaluation psychologist used in her arguments.

Her argument was essentially:
1. People that are largley able parents and generally normal people are being evaluated in a way that is almost always comprehensive in nature, leading to lots of billable hours for the Psychologist. People seem to end up in these custody battles largely due to a fear of the justice system and the idea that they will lose their child if they don't fight for the child in this manner, not because they aren't able to take care of their kids, or because they have some serious psychopathology.
2.Most of the tests used in a custody evaluation have no relevance in terms of parenting capacity. IQ tests are fairly standard but what is the relevance of IQ in terms of custody evaluation? Where is the research that shows that parents with higher IQ's make better parents? There is none. If the IQ test is simply used to rule out ******ation, does it really take a Psychologist to suspect such a thing? Couldn't an average person with literally no training suspect such a thing?

What I gathered from your comment erg was that more comprehensive testing is not neccesarily useful because the more information we have, the more inferences/judgements we have to make, and that often leads to worst decisions that are not evidence-driven.

As T4C pointed out, the answer to the first bolded portion is no. Maybe if you're talking specifically about folks who'd fall in the severe to profound range, sure, but particularly for mild, there's a lot of grey area, and I'd venture to say the majority of those folks wouldn't be readily identifiable to the "naked eye."

For the second portion, the points you've made are somewhat contradictory--having more data generally allows you to make more-informed and more reliable judgments, and is exactly what's meant by the term evidence-drive. Psychologists are asked to form and present professional opinions regarding a variety of factors; in custody evaluations, it might be which parent is better suited to the task, or if either is particularly ill-suited to being a parent. In those cases, as the goal is to present a fine-grained analysis, you're generally going to do a more thorough assessment than if you're working with a patient who readily identifies as depressed when presenting to a primary care doctor and who needs some brief, focused psychotherapeutic intervention. The former situation requires a lot of data-driven judgment making; the latter may just require a brief clinical interview. Also, in a legal context in particular, there are very, very strong incentives to present disingenuously, and it's in those situations in particular that relying solely on clinical judgment can be very problematic.
 
Not to change the subject of this thread, but that same argument you made there (in bold), is essentially the argument the anti custody-evaluation psychologist used in her arguments.

Most of the tests used in a custody evaluation have no relevance in terms of parenting capacity. IQ tests are fairly standard but what is the relevance of IQ in terms of custody evaluation? Where is the research that shows that parents with higher IQ's make better parents? There is none. If the IQ test is simply used to rule out ******ation, does it really take a Psychologist to suspect such a thing? Couldn't an average person with literally no training suspect such a thing?.

1. Why on earth would you deem the presence or absense of psychopathology be "irrelevant" to being a parent?! Do you think a caretakers behavior has no effect on the people people they care for?

2. Its a tool. One of many. No one is going to conclude you can t be a parent if your FSIIQ is 80-85. Thats not how the data is used and its a very unsophiscated way of looking at the complexities of psychological evaluations.
 
1. Why on earth would you deem the presence or absense of psychopathology be "irrelevant" to being a parent?! Do you think a caretakers behavior has no effect on the people people they care for?

Serious psychopathology is very relevant to parenting capacity but serious psychopathology isn't that common. I'm suggesting that the majority of people are showing up for custody evaluations not because they are facing issues that are any different from most normal folks, but they are showing up because the justice system promotes couples to fight for custody, and our society doesn't entice them to work outside of the justice system. If this is true, that is problematic.

Nobody is suggesting that a Psychologist will use an IQ test on its own to dictate a result but it becomes more likely to be used if psychopathology isn't found. The bigger question is, what relevance does it have exactly in terms of parenting capacity?

Furthermore, more data points are not useful for the majority of cases because serious psychopathology shouldn't be found nor assumed in most individuals. So if no psychopathology is found, does that mean that the parents are free to leave and make decisions in the best interest of their child? No. The Psychologist is left to make decisions based on what? Slightly increased levels of paranioa in one parent than the other? Assesments like IQ? Or they are left to make decisions based on an investigation (interviews/history/speculation) and the accusations levelled by each parent at each other?[/quote][/QUOTE]
 
Serious psychopathology is very relevant to parenting capacity but serious psychopathology isn't that common.

So, you would suggest ruling it out...how? A Ouija board board?

One tool psychologists use to rule in and rule out serious psychopathology is using psychological tests that are designed for this purpose.
 
Serious psychopathology is very relevant to parenting capacity but serious psychopathology isn't that common. I'm suggesting that the majority of people are showing up for custody evaluations not because they are facing issues that are any different from most normal folks, but they are showing up because the justice system promotes couples to fight for custody, and our society doesn't entice them to work outside of the justice system. If this is true, that is problematic.

I don't engage in custody evals, but I would wonder if psychopathology is actually much more common in the evals that have reached this stage.

And, if we are being strictly empirical, parents with higher IQ's tend to produce children with higher IQs and more success in life. It's group differences, which don't hold much weight when applied to individuals, as other variables will affect outcomes. But, to say it has no effect isn't quite accurate either.
 
Furthermore, more data points are not useful for the majority of cases because serious psychopathology shouldn't be found nor assumed in most individuals.

You need a graduate level stats class..
 
So, you would suggest ruling it out...how? A Ouija board board?

One tool psychologists use to rule in and rule out serious psychopathology is using psychological tests that are designed for this purpose.

How do you rule out most people who show up at your practice?
 
Top