Interrater Reliability and Psychologists?

[deleted722807]

[Deleted]

---

Members don't see this ad.

 

[WisNeuro]

Quick note on your first question. If that unnerves you, this career will be incredibly unnerving. Interrater reliability in research studies is one thing, how things actually work in the real world is another beast altogether. Politics, poor training, pure laziness, all play a role. Even putting PDs (I'm looking at you Borderline PD misdiagnosed as Bipolar like 80% of the time) aside for the moment, prepare to see misdiagnoses and /or over reliance on wastebasket diagnoses being the rule, and not the exception. If you have a problem with ambiguity in the patient chart, this may not be the specialty for you.

 

[PsyDr]

1) Using what instrument? MINI? SCID? DSM5 kappa scores for unstructured clinical interview vary between .7 and .2. If you're having problems with this, you should look into the sens/spec of various parts of the H&P. Hint: some of them are not great.

2) PsyDs are not necessarily therapists. CBT has many variations and many show significant efficacy. EMDR, which is nonsensical in its explanation, seems to have some efficacy despite its own "theory". There is an article which explains that if you took away the eye movement nonsense, the therapy would basically be a prolonged exposure one which has evidence base. Psychodynamic psychotherapy has some evidence base for some disorders, some showign similar efficacy to CBT. Psychoanalysis is not typically considered a treatment for pathology, but rather a method to learn and ameliorate traits. A decent simile might be that psychoanalysis is somewhat like plastic surgery. There are many theories of psychoanalysis including Freudian. Glen Gabbard is probably a better resource for someone in your stage of your career.

 

[Fan_of_Meehl]

I apologize in advance if this post is considered trolling or inappropriate for this forum. I am only a current M1 who is a long way from studying any psychiatry in school, and I can understand if this post is too amateurish for the professionals here. I have two questions, mainly about personality disorders. To clarify, I do not doubt the existence of personality disorders (it might be hard to pass my M2 exams if I do!), but I did have questions (not doubts, questions) on the accuracy of their diagnosis and the efficacy of treatments by providers who are not physicians.

My first question is this: are there studies on interrater reliability for personality disorders? I have found studies on PubMed that examine this for depression and bipolar disorder, but not for personality disorders. I became interested in this because of the fact that, leading up to the 2016 election, many articles were written by therapists about how Donald Trump has Narcissistic Personality Disorder. Allen Frances wrote an article last month, however, providing a dissenting opinion. I was a bit unnerved by how professionals could have such opposing views when looking at the same data (footage of Trump is out there for all to see).


My second question is this: how are therapists (PsyDs, LCSWs, DCSWs, LPCs) viewed by psychiatrists? Are one of those degrees viewed better than the others? Are their treatments viewed as scientific? An M2 told me that cognitive behavioral therapy is mentioned in the psych block at my medical school, and my understanding is that it has been rated well by studies and is the gold standard. There seem to be many therapists, however, who use EMDR for PTSD (which is not supported by evidence), have a psychodynamic orientation (which I do not believe in), or incorporate other unscientific aspects into their practice. Furthermore, it puzzled me that some psychiatrists also also seem to have training in (and an admiration for) psychoanalysis or psychodynamic therapy. I read a book by Martin Kantor, who was a professor at Mount Sinai, that mentioned the Freudian view that paranoia was repressed homosexuality (!) and a book on existential psychotherapy by Stanford's Irvin Yalom, that had claims such as "every relationship has a sexual component." These books, of course, are meant for professionals and I do not pretend to have understood them as well as a psychiatrist would have.

Again, I apologize if this post is inappropriate and thank you in advance for your time.

We live in chaos, but we seek order.

 

[thoffen]

My second question is this: how are therapists (PsyDs, LCSWs, DCSWs, LPCs) viewed by psychiatrists? Are one of those degrees viewed better than the others? Are their treatments viewed as scientific? An M2 told me that cognitive behavioral therapy is mentioned in the psych block at my medical school, and my understanding is that it has been rated well by studies and is the gold standard. There seem to be many therapists, however, who use EMDR for PTSD (which is not supported by evidence), have a psychodynamic orientation (which I do not believe in), or incorporate other unscientific aspects into their practice. Furthermore, it puzzled me that some psychiatrists also also seem to have training in (and an admiration for) psychoanalysis or psychodynamic therapy. I read a book by Martin Kantor, who was a professor at Mount Sinai, that mentioned the Freudian view that paranoia was repressed homosexuality (!) and a book on existential psychotherapy by Stanford's Irvin Yalom, that had claims such as "every relationship has a sexual component." These books, of course, are meant for professionals and I do not pretend to have understood them as well as a psychiatrist would have.

A good therapist is a good therapist, regardless of clinical education or theoretical orientation. Sure, some people do bias toward the more advanced degrees from more academic places when viewing a therapist. But some people also bias toward an MD degree, a Harvard residency, etc. I'm sure the average Harvard-trained MD is better than the average DO/DO-trained psychiatrist, but each has potential to be highly qualified or highly sketchy. Better to look at the individual.

EMDR is evidence-supported. It is not clear whether the EMDR technique itself is important to the therapy working, but it definitely works. And psychoanalysis and psychodynamic psychotherapy are also evidence-based, but the design of the studies are extremely limited, and it would be impractical to try to take a very long-term intensive therapy and try to pit it against some control group. Nonetheless, short-term dynamic therapies have evidence of the same nature of CBT studies and are valid treatment approaches.

If you know little about psychoanalysis and its history, and especially if you focus on antiquated psychoanalytic theory and take it far out of context, you will certainly come to the conclusion that it is nuts. Many look at it with a more neutral view and come to the same conclusion. But psychoanalysis as a field has an extremely rich literature and multiple divergent theoretical and practical stances. In modern institutes, it would be rare to see a person beholden to a strict theoretical approach. It is more practically viewed as a method of psychological exploration with attention to interpersonal processes, repetitious mental processes, and the unconscious. Its validity is really determined by its practitioners and participants.

 

[PsyDr]

This might be of interest:

Steinert, C., Munder, T., Rabung, S., Hoyer, J., & Leichsenring, F. (2017). Psychodynamic Therapy: As Efficacious as Other Empirically Supported Treatments? A Meta-Analysis Testing Equivalence of Outcomes. Am J Psychiatry, 174(10), 943-953. doi: 10.1176/appi.ajp.2017.17010057

 

[SeniorWrangler]

In my state, anyone can call themselves a therapist, and many do. The therapy they deliver is often an eclectic, generic "talk therapy" that gives the patient lots of emotional support but doesn't really give them any new tools or challenge them. I also suspect many many MD's and PhD's doing therapy end up doing much the same thing behind closed doors, regardless of their theoretical orientation.

If you look at the literature, empathy and interpersonal warmth accounts for a large part of the effect of psychotherapy. This makes it hard to judge the added benefit of using a specific therapeutic technique. Newer techniques do have the advantage of explicitly targeting specific symptoms with a limited time to work, making it easier for therapist and patient to decide if they're making progress.

 

[tr]

Indeed, I wonder why people still bother to do cross-modality comparison studies, since the literature has long established that the most reliable finding is the Dodo Bird effect - i.e., all psychotherapies are efficacious, and the differences in outcomes are almost entirely ascribable to the individual therapist and to the quality of the therapeutic alliance.

I admit to a personal bias towards CBT, partly because I like to see results quickly and psychodynamic approaches seem to take a good deal longer to work (though their proponents would say their effects are more lasting), but these are minor differences.

We should be trying to figure out what are the elusive qualities that make a good therapist, and whether they can be taught. There's some research on this (I particularly like Ablon and Jones Am J Psychiatry 2002; 159:775–783, which parsed specific therapeutic strategies and suggested that in fact the most effective therapists, regardless of what modality they believed themselves to be practicing, were using techniques most characteristic of CBT) but the pickings are still slim.

As to the OP's questions,

1) Actually interrater reliability is notoriously poor even for Axis I disorders!! though it's better when using a structured clinical interview like the SCID.
However, the situation isn't appreciably worse for Axis II. See
Attainment and maintenance of reliability of axis I and II disorders over the course of a longitudinal study - ScienceDirect

2) The therapist's degree is not the most important factor. Studies of this differ in that some find a modest positive effect of education and others find no difference in outcomes between highly trained and minimally trained practitioners as long as the therapy is manualized. No study has found a large, clinically significant effect of therapist educational level on therapy outcomes.

O'Donovan et al., Professional Psychology: Research and Practice, 36(1), 104-111
http://psycnet.apa.org/record/2005-01101-015

Van Oppen et al., J Clin Psychiatry 2010, 71(9):1158
http://commonweb.unifr.ch/artsdean/pub/gestens/f/as/files/4660/33127_092741.pdf

Michael et al., J Child Family Studies 2005, Volume 14, Issue 2, pp 223–236
Interventions for Child and Adolescent Depression: Do Professional Therapists Produce Better Results?

 

[Salpingo]

I apologize in advance if this post is considered trolling or inappropriate for this forum. I am only a current M1 who is a long way from studying any psychiatry in school, and I can understand if this post is too amateurish for the professionals here. I have two questions, mainly about personality disorders. To clarify, I do not doubt the existence of personality disorders (it might be hard to pass my M2 exams if I do!), but I did have questions (not doubts, questions) on the accuracy of their diagnosis and the efficacy of treatments by providers who are not physicians.

My first question is this: are there studies on interrater reliability for personality disorders? I have found studies on PubMed that examine this for depression and bipolar disorder, but not for personality disorders. I became interested in this because of the fact that, leading up to the 2016 election, many articles were written by therapists about how Donald Trump has Narcissistic Personality Disorder. Allen Frances wrote an article last month, however, providing a dissenting opinion. I was a bit unnerved by how professionals could have such opposing views when looking at the same data (footage of Trump is out there for all to see).


My second question is this: how are therapists (PsyDs, LCSWs, DCSWs, LPCs) viewed by psychiatrists? Are one of those degrees viewed better than the others? Are their treatments viewed as scientific? An M2 told me that cognitive behavioral therapy is mentioned in the psych block at my medical school, and my understanding is that it has been rated well by studies and is the gold standard. There seem to be many therapists, however, who use EMDR for PTSD (which is not supported by evidence), have a psychodynamic orientation (which I do not believe in), or incorporate other unscientific aspects into their practice. Furthermore, it puzzled me that some psychiatrists also also seem to have training in (and an admiration for) psychoanalysis or psychodynamic therapy. I read a book by Martin Kantor, who was a professor at Mount Sinai, that mentioned the Freudian view that paranoia was repressed homosexuality (!) and a book on existential psychotherapy by Stanford's Irvin Yalom, that had claims such as "every relationship has a sexual component." These books, of course, are meant for professionals and I do not pretend to have understood them as well as a psychiatrist would have.

Again, I apologize if this post is inappropriate and thank you in advance for your time.

I think you missed the point of the Allen Frances article. He agreed that while Trump may be a narcissist, he is functioning at too high of a level to have a personality disorder. Frances put together the DSM-IV, but has become critical of the way psychiatric diagnoses have become distorted and misused, so that mental illnesses have become dramatically overdiagnosed. In this case, he was upset that a PD was being used as justification to remove a seated president. While he feels that Trump is poorly qualified to be president, he should be judged on political merits and flaws, and not mental illness.

I believe that Trump is a mirror of the American soul, a surface symptom of our deeper societal disease. He may not be crazy, but we certainly were for electing him.

 

[Salpingo]

My second question is this: how are therapists (PsyDs, LCSWs, DCSWs, LPCs) viewed by psychiatrists? Are one of those degrees viewed better than the others? Are their treatments viewed as scientific? An M2 told me that cognitive behavioral therapy is mentioned in the psych block at my medical school, and my understanding is that it has been rated well by studies and is the gold standard. There seem to be many therapists, however, who use EMDR for PTSD (which is not supported by evidence), have a psychodynamic orientation (which I do not believe in), or incorporate other unscientific aspects into their practice. Furthermore, it puzzled me that some psychiatrists also also seem to have training in (and an admiration for) psychoanalysis or psychodynamic therapy. I read a book by Martin Kantor, who was a professor at Mount Sinai, that mentioned the Freudian view that paranoia was repressed homosexuality (!) and a book on existential psychotherapy by Stanford's Irvin Yalom, that had claims such as "every relationship has a sexual component." These books, of course, are meant for professionals and I do not pretend to have understood them as well as a psychiatrist would have.

Again, I apologize if this post is inappropriate and thank you in advance for your time.

Therapists are viewed by how well they provide therapy. In larger markets, therapists will typically carve out a niche for themselves in addiction, DBT/personality disorders, family therapy, etc. and reputation/expertise always trumps... err... beats out the letters following the name. The exceptions are neurocognitive disorders and sports psychology, which are psychologist dominated. I will admit my personal bias against PsyDs, since I'm always wondering why they didn't take the extra step to be PhDs, but thats me.

As for the psychodynamic emphasis in the field, thats mostly historical and geographical. I personally believe that the psychodynamic theory of mind is still useful and important, as is behavioralism, existentialism, humanism, etc. The reasons its still important can (and have) filled books, but I will mention that the therapists who make a point that psychoanalysis is unscientific and has no place in medicine seem to make some of the most mind numbingly stupid blunders. Same thing goes for the super dogmatic psychoanalysts.

 

[WisNeuro]

Indeed, I wonder why people still bother to do cross-modality comparison studies, since the literature has long established that the most reliable finding is the Dodo Bird effect - i.e., all psychotherapies are efficacious, and the differences in outcomes are almost entirely ascribable to the individual therapist and to the quality of the therapeutic alliance.

While I do agree with this, in general, I think the studies that show this haven't done as great of a job looking at specific disorders and comparisons. I'd wager that some certain disorders do, in fact, respond deferentially to certain treatments over others, above and beyond common factors.

 

[Shikima]

I read somewhere that mental health practitioners have some of the poorest IRR scores around.