Learning new assessments

[biomom]

As a part of my new job, I will be administering structured interviews. I’ve worked as a psychometrist before, so at first blush this seems like no big deal. I’ve started training for the first one though, and I’m finding the scoring to be more complicated than I imagined. I was given access to an online training which actually gave me more questions. Generally, I rely on the manual and read it carefully before training. I’m not able to read the manual in this case. This is causing me some anxiety. How do you all manage to learn something new while keeping a growth mindset? I want to obviously do this well.

---

Members don't see this ad.

 

[psych.meout]

If you're having difficulty learning part of the job, it would be best to speak with your supervisor, especially if you're going to be working under their license.

 

[erg923]

As a part of my new job, I will be administering structured interviews. I’ve worked as a psychometrist before, so at first blush this seems like no big deal. I’ve started training for the first one though, and I’m finding the scoring to be more complicated than I imagined. I was given access to an online training which actually gave me more questions. Generally, I rely on the manual and read it carefully before training. I’m not able to read the manual in this case. This is causing me some anxiety. How do you all manage to learn something new while keeping a growth mindset? I want to obviously do this well.

Why won’t your supervisor let you have the manual?

 

[Justanothergrad]

Why won’t your supervisor let you have the manual?

second question, why not buy the manual

 

[MCParent]

Yeah you 100% need to read the manual

 

[WisNeuro]

Yeah you 100% need to read the manual

Seconded, even if it's an update to an existing test (i.e., the new WAIS or WMS) you need to fully read the admin manual for administration, and a good deal of the technical manual if you are doing interpretation.

 

[biomom]

I’m in the process of trying to get the manual. The VA has an online training that seems like a summary of the manual. It’s an information dump and it’s not detailed enough about scoring criteria. I have searched on my own for how to buy the manual but have come up empty handed. I’m working remotely, so I’m not in the office to look at the manual. To be honest, they seemed surprised when I asked. I’ve always read and referred to the manuals when administering assessments. I feel pretty adamant about this. We’ll see what happens with my request.

 

[erg923]

I’m in the process of trying to get the manual. The VA has an online training that seems like a summary of the manual. It’s an information dump and it’s not detailed enough about scoring criteria. I have searched on my own for how to buy the manual but have come up empty handed. I’m working remotely, so I’m not in the office to look at the manual. To be honest, they seemed surprised when I asked. I’ve always read and referred to the manuals when administering assessments. I feel pretty adamant about this. We’ll see what happens with my request.

What is happening here? Are you student or a trainee with shifty oversight? Why are you "in the process of trying to get the manual?" You shouldn't be dong anything without this as student/trainee/psychometrist.

 

[Justanothergrad]

I’m in the process of trying to get the manual. The VA has an online training that seems like a summary of the manual. It’s an information dump and it’s not detailed enough about scoring criteria. I have searched on my own for how to buy the manual but have come up empty handed. I’m working remotely, so I’m not in the office to look at the manual. To be honest, they seemed surprised when I asked. I’ve always read and referred to the manuals when administering assessments. I feel pretty adamant about this. We’ll see what happens with my request.

what test is this about

and if it's so hard to get a manual... why that test?

 

[psych.meout]

what test is this about

and if it's so hard to get a manual... why that test?

And if you don't have access to the manual because you're working remotely, how are you supposed to administer it to patients if you can't be in the office?

 

[biomom]

It’s the CAPS-5. It’s a semi structured interview administered via telehealth. I have a few basic questions on how to rate certain items that I need to clear up. I will be sure to get my questions answered. I just prefer to read the manual or guide.

 

[biomom]

Maybe there is no manual in the true sense since it is an interview, but with the specificity about how certain things are rated I’d guess there is something.

 

[AbnormalPsych]

To be done well, in my opinion, The CAPS-5 requires very solid training and supervision. The interview technique and scoring thresholds take time to become even competent with. There is a lot of nuance that people without a trauma background do not realize. As a student/trainee, you cannot just pick this up and read it and go. Get supervision. Ideally, audio record and listen to the tapes with your supervisor in an ongoing manner as well.

It is a whole different ball game than a PCL.

 

[AcronymAllergy]

I don't actually know if there's a manual in the true sense for the CAPS-5; if so, the VA's national PTSD website should have whatever they've got. The ~four-hour training in TMS should walk through the basics of administration/scoring, and there's now an additional training that uses some type of patient simulation (I've not done it). I would agree that the first couple times going through it, especially if you aren't already relatively well-trained in structured interviewing and/or trauma assessment, you should seek out some supervision.

 

[cara susanna]

Agree with all of the above. The CAPS-5 doesn't have a manual that I know of, but the protocol itself has instructions that you can read over. The TMS course will be very helpful. I also do recommend supervision for at least the first few cases.

 

[biomom]

I agree with supervision. I have some experience assessing trauma, and although I’m not a student anymore, I’m a trainee. I’m most concerned with intensity ratings and overall severity. The TMS course has some good parts, but it would be nice to have in a book format. I’ve started taking pictures of each screen to print out as a reference.

 

[AcronymAllergy]

I agree with supervision. I have some experience assessing trauma, and although I’m not a student anymore, I’m a trainee. I’m most concerned with intensity ratings and overall severity. The TMS course has some good parts, but it would be nice to have in a book format. I’ve started taking pictures of each screen to print out as a reference.

I took notes as I went through the TMS course, as I definitely didn't catch/remember all the information on my own. They do include some specific guidelines on the different ratings. Screen printouts could also be great, but I'd probably still need my own additional notes on them.

 

[biomom]

Thanks for the tip. I’ll add notes as well.

I took notes as I went through the TMS course, as I definitely didn't catch/remember all the information on my own. They do include some specific guidelines on the different ratings. Screen printouts could also be great, but I'd probably still need my own additional notes on them.

 

[Justanothergrad]

To be done well, in my opinion, The CAPS-5 requires very solid training and supervision. The interview technique and scoring thresholds take time to become even competent with. There is a lot of nuance that people without a trauma background do not realize. As a student/trainee, you cannot just pick this up and read it and go. Get supervision. Ideally, audio record and listen to the tapes with your supervisor in an ongoing manner as well.

It is a whole different ball game than a PCL.

I mean, I agree with you to some degree....



And at the same time... Bovin et al (2016)

 

[cara susanna]

I mean, I agree with you to some degree....



And at the same time... Bovin et al (2016)

Sorry, not quite getting this. Are you disagreeing that the CAPS-5 is a whole different ball game from the PCL-5?

 

[Justanothergrad]

Sorry, not quite getting this. Are you disagreeing that the CAPS-5 is a whole different ball game from the PCL-5?

In terms of process, sure, in terms of ultimate outcomes with verified cut-scores? na. not really. Especially with 2 minutes of clarifying questions by the clinician.

 

[WisNeuro]

In terms of process, sure, in terms of ultimate outcomes with verified cut-scores? na. not really. Especially with 2 minutes of clarifying questions by the clinician.

In general, yeah. A good trauma clinician essentially treats the PCL as a semistructured interview. Anyone who relies on the self-report only really just doesn't know what they are doing. Used it both my thesis and dissertation. In self-report, almost 50% endorsed the flashbacks question, upon follow-up questioning, I believe it was down to 2 people in one of my studies.

 

[Justanothergrad]

I finished the first TMS training and then went back and took pictures of all the early slides I missed. I also typed notes. There are 2 examples of scoring intensity and they warn not to use these examples. I don’t score assessments based on my gut feelings about whether something is mild or moderate etc. There should be crystal clear definitions and examples in a manual to refer to. I’m not going to guess what the intensity level is of a problem that someone struggles with “a lot”. I have searched high and low for clear guidelines, and have not heard back from my boss or the research coordinator. I don’t want to be difficult, but I’d like the clarification. Maybe I’m overthinking it.

personal experience and resource is subjective, thus concrete rules are difficult to impossible to establish. Intensity is, ultimately, a clinical judgement

 

[AbnormalPsych]

I mean, I agree with you to some degree....



And at the same time... Bovin et al (2016)

I don't follow.

In terms of process, sure, in terms of ultimate outcomes with verified cut-scores? na. not really. Especially with 2 minutes of clarifying questions by the clinician.

Yes, I was talking about the process. Not outcomes.

 

[cara susanna]

If you look at the CAPS questions, on the bottom it tends to give concrete standards for what it means by intensity. It's under "Key rating dimensions."

Of course, there is still some clinical judgment involved.

 

[Justanothergrad]

I don't follow.

Yes, I was talking about the process. Not outcomes.

My point is that if there is a case of equifinality in outcomes, I wonder the degree to which differences in process produce meaningful choice points. It's outside of the thread's original purpose, but relevant.

 

[mdwpsyd]

And at the same time... Bovin et al (2016) ...

In terms of process, sure, in terms of ultimate outcomes with verified cut-scores? na. not really. Especially with 2 minutes of clarifying questions by the clinician. ...

My point is that if there is a case of equifinality in outcomes, I wonder the degree to which differences in process produce meaningful choice points. It's outside of the thread's original purpose, but relevant.

You referred to:

Bovin, Michelle J., Brian P. Marx, Frank W. Weathers, Matthew W. Gallagher, Paola Rodriguez, Paula P. Schnurr, and Terence M. Keane. “Psychometric Properties of the PTSD Checklist for Diagnostic and Statistical Manual of Mental Disorders-Fifth Edition (PCL-5) in Veterans.” Psychological Assessment 28, no. 11 (November 2016): 1379–91. APA PsycNet

If I understand you correctly, you are saying that if one diagnoses PTSD based on PCL-5 scores of 31 or higher (with 2 minutes of clarifying questions), one will achieve equivalent accuracy as compared to diagnoses based on CAPS-5 results.

You might be right ...

• if one is assessing U.S. military veterans who are receiving VA health care services; and
  
  
• if the veterans one assesses do not consider at all the potential impact of one's assessment results on a current or near-future disability compensation claim for PTSD; and
  
  
• if the base rate (prevalence) of PTSD at one's clinic is similar to that in the Bovin, et al. (2016) study (61%).

What do you think?

 

[Justanothergrad]

You referred to:

Bovin, Michelle J., Brian P. Marx, Frank W. Weathers, Matthew W. Gallagher, Paola Rodriguez, Paula P. Schnurr, and Terence M. Keane. “Psychometric Properties of the PTSD Checklist for Diagnostic and Statistical Manual of Mental Disorders-Fifth Edition (PCL-5) in Veterans.” Psychological Assessment 28, no. 11 (November 2016): 1379–91. APA PsycNet

If I understand you correctly, you are saying that if one diagnoses PTSD based on PCL-5 scores of 31 or higher (with 2 minutes of clarifying questions), one will achieve equivalent accuracy as compared to diagnoses based on CAPS-5 results.

You might be right ...

• if one is assessing U.S. military veterans who are receiving VA health care services; and
  
  
• if the veterans one assesses do not consider at all the potential impact of one's assessment results on a current or near-future disability compensation claim for PTSD; and
  
  
• if the base rate (prevalence) of PTSD at one's clinic is similar to that in the Bovin, et al. (2016) study (61%).

What do you think?

Yeh, I agree with you - norms and studies on an instrument's utility are situationally specific and consideration of their use requires awareness of factors associated with their influence (e.g., base rate, secondary motivation/validity of data, race/ethnicity/age/gender/education/etc etc etc.). My broader point was that the CAPS-5 isn't some magical force of PTSD diagnosis that carries with it some form of validity beyond others when ultimately it relies on some of the same determinations which must be made clinically as any other structured or semi-structured interview because we identify 'impact'. A bad clinician with it will perform poorly; a good clinician with a PCL will perform well... because of the clinician judgement component.

I'm not a fan of over-diagnosis and I focus a lot of my work on translating scales across groups to ensure situationally appropriate use. I enjoyed your 2011 paper on C&P btw - cited it a few times in papers as I recall. Your #2 point is a bit broader than the issue of norms in #1/#3 and gets to the systematic process of C&P, rather than just validity per se.

 

[mdwpsyd]

Cool. Sounds like we're on the same page. And thank you so much for your kind words! :0) ~ Mark