Third party observers

[R. Matey]

I read recently that some consider third party observation to be seriously threatening to the validity of the results of psych testing so I thought I'd punt the question to SDN. Is it? Is third party observation done at your site? What's the research say and not say about it? I never sat in on evals with real patients during training, but I would imagine this would make some evaluations trickier to learn, particularly with children.

---

Members don't see this ad.

 

[PsyDr]

Not how tests are normed
Affects results, even when unknown

I won’t work under those circumstances.

 

[neurotic_cow]

NAN put out an official statement regarding the presence of third party observers (TPO) which discusses this issue in depth.

Aside from the fact that our tests are not normed on having people take tests with a third party observer, there's concerns about test security, how the TPO may impact test behavior and performance, amongst other issues. There's even some research to suggest that even A/V recording of testing may impact performance.

I worked at a practicum site once where my supervisor forced me to have a daughter (adult) sit in on testing with her mother citing some sort of concern about her distress during testing. The entire time I was testing, the daughter sat on her phone which made noises (had to ask her to turn it off multiple times), noisily opened a bag of chips and munched away, and at one point interrupted testing to argue with me about how the test was being administered. It was a distraction for not only myself (impacting standardization of administration) as well as for the patient, essentially invalidating the evaluation. I realize most TPO may not be that blatantly disruptive but it's always a possibility and it was a frustrating experience on multiple levels.

Long story short: It's a bad idea. I refuse to do it and seriously question the competence of people who do it or allow it.

 

[WisNeuro]

Outside of training circumstances, I never allow TPO. It's coming up more and more in IMEs these days, but in almost ever situation we've been able to convince the judge to not allow it, but who knows how long that lasts. In a case in which the judge does allow it, I'll simply withdraw from the case.

In terms of training, we will allow it, but we'll try to minimize the effects of it as best we can. I'll pick and choose the cases in which I directly observe someone giving the tests.

 

[AcronymAllergy]

Agreed with the above, I do not agree to perform evaluations with TPO outside of training situations. And in training situations, we do what we can to minimize the impact of observers (and you could potentially make the logical argument that a trainee or supervisor, given their knowledge of test design and administration, may be a less-disruptive observer than an untrained individual), select cases for which it may be most appropriate (and end observation if it's interfering with testing), and understand the influence it will have on results.

Edit: and given borne_before's post below, I should clarify that I only perform adult evals. I can see how there might be other factors at play in arriving at a decision when evaluating kiddos.

 

[borne_before]

Outside of a training circumstances, I never really allow third party observations.

However, there is some caveats. I am a male - and have had several parents with histories of sexual abuse be less than stoked about me testing their minor child. In those situations, I will test in a room with mirrored window and allow them to observe.

If a parent is doing it because they want to exert some control over the process - that's gonna be a big "no" and a great way to weed them out.

When I start my private practice, I will probably set up a baby camera with the mic/sound disabled so that parents can observe from the waiting room if they so wish to observe things. I'll take care to make sure the resolution is low enough to ensure test security.

 

[R. Matey]

Great discussion, folks. Thanks! A few follow up questions. I promise I don't have any motive aside from general curiosity:

Not how tests are normed
Affects results, even when unknown

Any thoughts on effect size here? Like would an observer skew the results more than say a hot testing room, needing extra bathroom breaks, etc? These are regular issues in educational testing.

NAN put out an official statement regarding the presence of third party observers (TPO) which discusses this issue in depth.

What I read was an update to that statement. I thought it was a bit skimpy on the research though. But maybe there's a systematic review or meta-analysis that I missed...

Outside of training circumstances, I never allow TPO. It's coming up more and more in IMEs these days, but in almost ever situation we've been able to convince the judge to not allow it, but who knows how long that lasts. In a case in which the judge does allow it, I'll simply withdraw from the case.

In terms of training, we will allow it, but we'll try to minimize the effects of it as best we can. I'll pick and choose the cases in which I directly observe someone giving the tests.

So it sounds like that training would be an alright circumstance. We used confederates during my training both in coursework and at my testing prac before real patients. I did have supervisors sit in with me for difficult cases a few times. I could why IMEs or forensic cases would have stakeholders who want in, but are there other circumstances would TPOs would be a non-starter?

 

[PsyDr]

Effect sizes vary between medium to large (e.g., d=-0.24). Methods of TPO vary from parental presence, direct observation, and surreptitious audio recording included.

I have no idea how a hot room, or bathroom breaks would affect testing. Why wouldn't people get bathroom breaks? That's weird.

 

[WisNeuro]

Effect sizes vary between medium to large (e.g., d=-0.24). Methods of TPO vary from parental presence, direct observation, and surreptitious audio recording included.

I have no idea how a hot room, or bathroom breaks would affect testing. Why wouldn't people get bathroom breaks? That's weird.

I had an opposing attorney who had some weird issues and tried to get out of my evaluation saying that my consent form didn't allow his client "necessary and special accomodations according to his Dr's orders." The accommodations he wanted? A 5 minute break every hour. Person who retained me had to go back and forth, and ultimately got a judge to order the eval forward. 1) My consent form in no way bars that. It does have some language pertaining to accommodations that I do not feel comfortable/competent/have the necessary equipment to do for legal cases (e.g., ASL, braille materials). And 2) I had to explain that every single claimant and client that I evaluate is asked if they need a break about every half an hour, so by definition, it's not an accommodation if its a regular part of the evaluation process. If anything, his Dr's orders required fewer breaks than I would normally offer.

 

[Ollie123]

For training purposes, we usually relied on very unobtrusive observation (mirror/video monitor) versus in the room, then rehearsed with one another, then practiced with "fake patients". I think our neuropsych prof's son had probably been WISC'd 200 times, which is not ideal from a test security standpoint, but was great for training. He was well-coached and she quite successfully transferred multiple decades worth of experience in ways people cause problems during testing. He also seemed to derive tremendous joy from parent-approved efforts to make other adults freak out.

I didn't do much formal assessment past grad school. Intake evals and the like would occasionally involve a support person. Even that was a mixed bag whether it was helpful versus harmful. I can't imagine trying to do formal testing. That said, for reasons R.Matey already laid out I think there is always an aspirational element to test validity. Perfect standardization just isn't possible, so it becomes a give and take to get as close as possible within the parameters set by the context.

 

[JeyRo]

I'm fascinated by people's reactions. Maybe it's about perspective. I'm ABPP'd in gero and I've been doing basically literally nothing but geropsychology (with a very assessment-heavy emphasis in my practice) for nearly the last 20 years and I think that "third party observers" are fine, and sometimes actually helpful when it's handled properly.

Example that comes to mind is when I do dementia diagnostic evals, and I'm doing neuropsych testing with the patient - very, very often I have the caregiver (wife, daughter, etc.) sit behind the patient while I do the testing. It makes the patient often far more comfortable and makes managing test anxiety and other issues far easier in many cases.

 

[WisNeuro]

I'm fascinated by people's reactions. Maybe it's about perspective. I'm ABPP'd in gero and I've been doing basically literally nothing but geropsychology (with a very assessment-heavy emphasis in my practice) for nearly the last 20 years and I think that "third party observers" are fine, and sometimes actually helpful when it's handled properly.

Example that comes to mind is when I do dementia diagnostic evals, and I'm doing neuropsych testing with the patient - very, very often I have the caregiver (wife, daughter, etc.) sit behind the patient while I do the testing. It makes the patient often far more comfortable and makes managing test anxiety and other issues far easier in many cases.

In my inpatient work, I've found the opposite to be true. Patients often look to others in the room when they are unsure, and are far less likely to guess, even when prompted. I would only do a dementia eval with TPO when absolutely necessary. Anxiety is pretty easy to manage in the interview/informed consent process to add in the known empirical issues with TPO.

 

[JeyRo]

In my inpatient work, I've found the opposite to be true. Patients often look to others in the room when they are unsure, and are far less likely to guess, even when prompted. I would only do a dementia eval with TPO when absolutely necessary. Anxiety is pretty easy to manage in the interview/informed consent process to add in the known empirical issues with TPO.

I've for sure had cases where it doesn't work to have the caregiver in there - and placement in the room is actually important. It never works when the family member (spouses and child caregivers are notorious for this) is present and facing the patient, they invariably give cues and mess up the testing (hence having the caregiver sit behind the test subject).

Basically I've had arrangements like this work well enough times whereby I think making hard and fast rules about so-called TPO is a bit hasty in my opinion. Standardization ("empirical issues") are really important, for sure - but I see the other side of that coin is maximizing rapport and effort from the client, so you have to be willing to flex on standardization at times in order to do that. Just my opinion though - I'm no neuropsychologist.

 

[WisNeuro]

I've for sure had cases where it doesn't work to have the caregiver in there - and placement in the room is actually important. It never works when the family member (spouses and child caregivers are notorious for this) is present and facing the patient, they invariably give cues and mess up the testing (hence having the caregiver sit behind the test subject).

Basically I've had arrangements like this work well enough times whereby I think making hard and fast rules about so-called TPO is a bit hasty in my opinion. Standardization ("empirical issues") are really important, for sure - but I see the other side of that coin is maximizing rapport and effort from the client, so you have to be willing to flex on standardization at times in order to do that. Just my opinion though - I'm no neuropsychologist.

I guess I've just only ever rarely had instances where this has been a problem. In the maybe once a year I defer testing due to anxiety, it's usually bad enough that I sincerely doubt a family member being present would help. If anything, in these cases, it would likely hinder the evaluation even more as oftentimes these patients differ in the level of insight about their cognitive problems as opposed to their spouse/children, and thus having these issues on full display in front of that person I'd have to imagine would only increase that anxiety/frustration.

 

[smalltownpsych]

I vote for a general no on TPO other than some training situations where effects would be low. We typically practiced our adminstration on family members and other students and faculty. I think I did observe a six year old kid do an assessment once as a practicum student. This was the kid that tried to run away when I took her to get a snack during a break in the testing. If I had lost that kid that would have really affected the results!

 

[borne_before]

I vote for a general no on TPO other than some training situations where effects would be low. We typically practiced our adminstration on family members and other students and faculty. I think I did observe a six year old kid do an assessment once as a practicum student. This was the kid that tried to run away when I took her to get a snack during a break in the testing. If I had lost that kid that would have really affected the results!

God damn elopers!!! They can be so fast lol.

 

[singasongofjoy]

Isn't the answer to like 90% of things in psych "it depends?" certainly on which assessment, situational factors related to client (e.g., age, anxiety/comfort level, cognitive capacity, or... related to above... how fast can they run, lol)? There are certainly times I want/bring in observer to know whether what I am seeing is actually representative behavior if something seems incongruent with info given in interview / obtained elsewhere. even with minors it's only exceptional circumstances I'd have observation without asking the client first about their preference (and if they first want third party to remain in room or observation room I check in again a bit later) or telling them that so-and-so will be in the observation room for a bit (usually filling out questionnaires / report measures with a trainee also present). If you have an observation room or similar setup seems you could maintain integrity of test questions by not having the speakers enabled and thoughtfulness in positioning you, client, and stimuli.

 

[ClinicalABA]

With toddlers, parents/caregivers present in room (testing protocols/norming account for this). Generally kids over 4 y.o. I see alone, with caregiver in waiting room for cog/language/exec functioning norm-referenced stuff (door is open- I am never alone in closed room with a child). Parents join for some criterion-referenced tests (e.g. ADOS-2) as specified in testing protocols.

 

[Spydra]

As an intern and previously as a practicum student I observe all the time. I refused to train in environments that did not allow this because books and videos are not sufficient for learning. Now when I am conducting the eval the only circumstances in which I have a third party observer is if I have safety concerns and want security present or if its a child that I suspect might behave in an inappropriate manner and I want the parent present to keep them on track.