Statistics behind patient satisfaction surveys

[BAM!]

I'm trying to understand patient satisfaction scores and their relevance. (yeah I know... they're irrelevant, yada yada).

For example, Press Gainey takes 30-50 surveys and assigns a score to a physician. According to ACEP,

"for the ED as a whole 30-50 results could provide as low as a 50-55% confidence interval (a flip of a coin to decide whether the results are valid). In order to create the generally accepted 95% confidence interval, 175-225 surveys would be necessary for a scaled (1-5 for example) survey, which is the methodology that the company uses."

ref - https://www.acep.org/patientsatisfaction/

Seems like perhaps an individual might need a larger amount than 30-50 surveys. Can someone comment on how to calculate the number of surveys needed to provide a 95% CI for an individual? It's been a while since i've taken biostat.

---

Members don't see this ad.

 

[erdoc00]

You're asking irrelevant questions.. how it's calculated or if it's even valid are meaningless in comparison to the fact that this is what the administrators and government have chosen as the metric.

Follow the money trail and you'll get the answers. There is no logic or statistics involved.

 

[Arcan57]

I'm trying to understand patient satisfaction scores and their relevance. (yeah I know... they're irrelevant, yada yada).

For example, Press Gainey takes 30-50 surveys and assigns a score to a physician. According to ACEP,

"for the ED as a whole 30-50 results could provide as low as a 50-55% confidence interval (a flip of a coin to decide whether the results are valid). In order to create the generally accepted 95% confidence interval, 175-225 surveys would be necessary for a scaled (1-5 for example) survey, which is the methodology that the company uses."

ref - https://www.acep.org/patientsatisfaction/

Seems like perhaps an individual might need a larger amount than 30-50 surveys. Can someone comment on how to calculate the number of surveys needed to provide a 95% CI for an individual? It's been a while since i've taken biostat.

The answer is going to depend on the standard deviation within the population sampled which PG doesn't reveal. But it doesn't really matter. The distribution of the raw score is so narrow that conversion to percentile rank (which is what c-suite cares about) guarantees that a couple of point difference in score will raise or drop one or more quartiles.

 

[shookwell]

The only way to increase your press ganey scores is to harm your patients

 

[southerndoc]

One of these days physicians will unite and agree not to go by them.

 

[GeneralVeers]

Just wait....pretty soon CMS will begin tracking our scores nationally. They will be attached to individual physicians on published on a national "quality database". Our hireability will depend on how quartile ranking compare nationally to other physicians.

 

[shookwell]

One of these days physicians will unite and agree not to go by them.

.
Both of the hospitals I work at (outside of locums) do not track press ganey scores because they are not useful and the ED docs and ED leadership were able to stand up to administration on this. One of them does send out the surveys but the results are only published as aggregate data and individual doctors' scores are not tracked.

 

[sum dude]

I sat through a PG lecture from one of their head guys, he basically said "they aren't really stat. significant" and should only be used as a "self-improvement exercise." When 90% puts you at 66th percentile, you know something is up.

Only don't look for help from the govt. It's an easy way to decrease reimbursement, and it sounds good to patients. I heard an anecdote of when CMS met with some leaders in EM. They told CMS the HCAPS surveys were flawed, explained ways to decrease the amount of questions to improve the survey to make it more useful and stat. significant. They looked right at them and told them, "we don't care if they are statistically significant...we just want patients to have a way of recording their experience." The survey has since then not changed (though they no longer use pain question to determine reimbursement.

Basically, cards of stacked against us. The sooner you realize your government/hospital admin don't care about stat. significant info, the better you will do. It appears we are in the bargaining portion of grief...already had denial and plenty of anger. Go ahead move on to the depression part, then accept, b/c unfortunately we are screwed.

 

[BAM!]

So if someone is fired because of low patient satisfaction scores, is there grounds for wrongful termination?

 

[BAM!]

The answer is going to depend on the standard deviation within the population sampled which PG doesn't reveal. But it doesn't really matter. The distribution of the raw score is so narrow that conversion to percentile rank (which is what c-suite cares about) guarantees that a couple of point difference in score will raise or drop one or more quartiles.

how would you calculate it if you knew the SD and had the raw data?

 

[Psai]

I don't get it. Seems like the majority of people are there for intoxication or the worried well who want antibiotics for sniffles or some other minor complaint that they should have went to their primary for but didn't want to wait for an appointment. A lot of them were already diagnosed and treated but haven't followed up with appointments or didn't fill prescriptioms.

I don't understand the utility of asking patients whether they were satisfied with their care. It's not my fault if labs take forever to come back or if the specialist doesn't come down for hours.

 

[WilcoWorld]

I don't get it. Seems like the majority of people are there for intoxication or the worried well who want antibiotics for sniffles or some other minor complaint that they should have went to their primary for but didn't want to wait for an appointment. A lot of them were already diagnosed and treated but haven't followed up with appointments or didn't fill prescriptioms.

I don't understand the utility of asking patients whether they were satisfied with their care. It's not my fault if labs take forever to come back or if the specialist doesn't come down for hours.

Moreover, admitted patients - you know, the ones with emergencies - aren't surveyed. It's like scoring the staff at In 'N Out burger based on the vegetarians who wandered into the store while they were looking for Panera.

 

[Arcan57]

One of these days physicians will unite and agree not to go by them.

No.

how would you calculate it if you knew the SD and had the raw data?

You'd just need to plug it into a power calculator. Based on some calculations I've run messing with the SD of the sample the numbers you were quoting are within the range of reason. But again, that's for raw score and since the majority of hospitals are in very close proximity on raw score the percentile curve is incredibly steep for raw scores in the 80s.

 

[Psai]

Moreover, admitted patients - you know, the ones with emergencies - aren't surveyed. It's like scoring the staff at In 'N Out burger based on the vegetarians who wandered into the store while they were looking for Panera.

Wait but those are the only people that really matter

I'm tired of dealing with the not sick at all "why don't you just look it up in the computer" crowd

 

[GeneralVeers]

My new group in Vegas has just started with the Press Ganey stuff. I'm mystified as to why they didn't have to deal with it years ago. I see e-mail after e-mail of docs going over statistics, and making well-reasoned arguments against it. As sum dude explained, they are in the bargaining stage still. They believe that a well-reasoned argument will change the minds of the CMG and the hospital system.

Fortunately I'm in the acceptance stage. I understand that the math is flawed, and that what we are measuring is irrelevant (to us) anyway. However hospital admin wants it in order to get their 7 figure bonuses. The best way to deal with it, is to pay lip service to "improving patient satisfaction" while at the same time making it minimally punitive on the docs.

 

[Birdstrike]

I'm trying to understand patient satisfaction scores and their relevance. (yeah I know... they're irrelevant, yada yada).

For example, Press Gainey takes 30-50 surveys and assigns a score to a physician. According to ACEP,

"for the ED as a whole 30-50 results could provide as low as a 50-55% confidence interval (a flip of a coin to decide whether the results are valid). In order to create the generally accepted 95% confidence interval, 175-225 surveys would be necessary for a scaled (1-5 for example) survey, which is the methodology that the company uses."

ref - https://www.acep.org/patientsatisfaction/

Seems like perhaps an individual might need a larger amount than 30-50 surveys. Can someone comment on how to calculate the number of surveys needed to provide a 95% CI for an individual? It's been a while since i've taken biostat.

You're making this way too hard. The statistical smokescreens used to obscure the truth about the "patient satisfaction" craze are irrelevant. But the motivation behind patient satisfactions scores themselves isn't "irrelevant." It's incredibly relevant to those that control physicians, nowadays. Chasing patients satisfaction scores is about marketing and money, for administrators and corporate. Period.

Don't get bamboozled by some pseudo-doctor turned administrator, giving a presentation trying to make the concept of patient satisfaction palatable to clinical types by candy-coating it with words like "data," "confidence intervals," and "percentiles." These are persuasion approaches, using techniques to pacify and hypnotize clinical types to comply with corporate culture.

It's no different than when the manager at Applebee's hands out comment cards. It's not scientific. It's not designed to help the waiters. It's not designed to create healthier food choices for the "customers," and reduce heart disease risk amongst the diners. "Percentiles" and "95% confidence intervals" don't mean bunk in this setting. They're just there to make you think there's some scientific basis resembling the science of Medicine, behind it all, so you'll accept it.

They want the unattainable, 100% percent all the time, from everybody, everywhere. They want your blinders on, chasing the rabbit always just beyond your reach. They don't care when science, "medicine" or ethics, clash with the worst impulses of human nature, such as instant gratification, drug addiction and cravings, and demands for irrational over-testing and zero-miss outcome-perfection.

They don't care that when irrational demands for bad and inappropriate medical (the expected unneeded-CT, the inappropriate antibiotics or narcotics demanded) are met with proper and ethical medical care, scores (and cash flow) will suffer. They want that inappropriate care to continue unabated, and they need your signature on it, to give them plausible deniability. It generates monetary charges and profits. And after all, a "Doctor" signed off on it, as "necessary."

In a fast food joint the customer "is always right." To them, that concept can be applied properly to medical settings. To them, the Hypocratic Oath is as outdated as the wooden teeth worn worn by our founding fathers. They'll never concede that it's wrong, unethical and bad medicine, although they absolutely know it to be true. They don't care, as long as profits are greater next quarter. Period. Their money, their bottom line are everything and the only thing. This comes as a shock to doctors. This is not at all surprising to those in the corporate world, which has cannibalized Medicine.



JAMA: Higher patient satisfaction is associated with 1-higher overall health care and prescription drug expenditures, and 2-increased mortality http://ja.ma/2c4Iacm

 

[deleted109597]

I got a report back that had a whopping 2 surveys. 1 was a 5, the other was a 1. Guess how little I cared about what it, or any other press ganey has ever said?

 

[Birdstrike]

I got a report back that had a whopping 2 surveys. 1 was a 5, the other was a 1. Guess how little I cared about what it, or any other press ganey has ever said?

Somebody cares. Not you, but the guys paying for the surveys.

 

[Arcan57]

My new group in Vegas has just started with the Press Ganey stuff. I'm mystified as to why they didn't have to deal with it years ago. I see e-mail after e-mail of docs going over statistics, and making well-reasoned arguments against it. As sum dude explained, they are in the bargaining stage still. They believe that a well-reasoned argument will change the minds of the CMG and the hospital system.

Fortunately I'm in the acceptance stage. I understand that the math is flawed, and that what we are measuring is irrelevant (to us) anyway. However hospital admin wants it in order to get their 7 figure bonuses. The best way to deal with it, is to pay lip service to "improving patient satisfaction" while at the same time making it minimally punitive on the docs.

They probably didn't have to deal with it because the profit margin was high enough they didn't have to care or they had no competition. As profit margins or market share shrink, less variability is tolerated. We've (thus far) successfully fought back against secret shoppers so that leaves patient sat surveys as the easiest way to validate that what they expect of us is being done. And if you're not validating then you don't know if the behavior is hard-wired. And if you don't know if the behavior you're being paid to produce is hardwired then that fat bonus check you're betting on may slip through your grasp.

 

[Apollyon]

They probably didn't have to deal with it because the profit margin was high enough they didn't have to care or they had no competition. As profit margins or market share shrink, less variability is tolerated. We've (thus far) successfully fought back against secret shoppers so that leaves patient sat surveys as the easiest way to validate that what they expect of us is being done. And if you're not validating then you don't know if the behavior is hard-wired. And if you don't know if the behavior you're being paid to produce is hardwired then that fat bonus check you're betting on may slip through your grasp.

Maybe it's because I have a few beers in me, but, huh??

 

[sum dude]

Maybe it's because I have a few beers in me, but, huh??

I think what he's trying to say is that market forces are forcing groups to acknowledge Pt Satisfaction. And "market forces" are artificially set by the government/CMS, who is trying to decrease reimbursement, and this is an easy way to do it. They tell big hospital groups (which are bigger now thanks to consolidation) what they want, the Big hospitals tell indvidual hospitals what they want, and your group has to do it or lose contract. Hence, stat. not significant PG's become a big deal. Hence, 20 pt satisfaction training's a year, and more bitching on doctor forums about our loss of autonomy and stupid pt satisfaction surveys...

Govt policies-->CMS-->(copied by) -->insurances)-->hospital groups--> Individual hospitals-->Groups--> physicians.

We're the end of the tail.

 

[GeneralVeers]

It's all nonsense so I ignore it, and practice how I feel appropriate. At the moment there are enough jobs desperate for doctors, that I can always leave.

Everywhere it's the same: Wear white coats, hand out business cards, use AIDET, say "we are going to take VERY GOOD care of you", have a doctor see every patient, etc. etc.

It's all total nonsense. I speak to hospital administrators like I would any delusional psych patient. I acknowledge their delusion, don't try to talk them out of it, change the discussion to something else and move on.

 

[Birdstrike]

️️️️️

It's all nonsense so I ignore it, and practice how I feel appropriate. At the moment there are enough jobs desperate for doctors, that I can always leave.

Everywhere it's the same: Wear white coats, hand out business cards, use AIDET, say "we are going to take VERY GOOD care of you", have a doctor see every patient, etc. etc.

It's all total nonsense. I speak to hospital administrators like I would any delusional psych patient. I acknowledge their delusion, don't try to talk them out of it, change the discussion to something else and move on.

 

[RustedFox]

Don't get me started on the "wear white coats" thing.

I particularly hate that nonsense. Its not 1986. White coats are fomites. All the commonwealth nations have "banned" them. "Nurse administrators" wear them to feel special. We have splash-resistant, stain-resistant, and microbe-resistant fabrics to make form-fitting and comfortable scrubs out of.

... but they want you to wear that freaking white coat.

 

[Birdstrike]

Don't get me started on the "wear white coats" thing.

I particularly hate that nonsense. Its not 1986. White coats are fomites. All the commonwealth nations have "banned" them. "Nurse administrators" wear them to feel special. We have splash-resistant, stain-resistant, and microbe-resistant fabrics to make form-fitting and comfortable scrubs out of.

... but they want you to wear that freaking white coat.

They're great to wear. So you have pockets. For individually wrapped bacons. And pop tarts. For snack time.

And they deflect (and collect) vomit, blood, and puss.

Very useful.

 

[RustedFox]

"Pus" has one "s".

At no time do I want to sit down in a chair and lose everything that I put in those pockets.

Oh, and fomites.

 

[Arcan57]

They're great to wear. So you have pockets. For individually wrapped bacons. And pop tarts. For snack time.

And they deflect (and collect) vomit, blood, and puss.

Very useful.

Tell me more about this individually wrapped bacon.

 

[goodoldalky]

"Pus" has one "s".

At no time do I want to sit down in a chair and lose everything that I put in those pockets.

Oh, and fomites.

Maybe he meant puss. Which to me means something different. Maybe he's trying to collect it with his white coat.
Bird please clarify.

 

[RustedFox]

Ha. Now Mrs. RustedFox doesn't want me to wear a white coat if it collects puss.

 

[GeneralVeers]

In anatomy lab in medical school someone put a puss in someone else's labcoat pocket.

 

[DissocativFugue]

Ha. Now Mrs. RustedFox doesn't want me to wear a white coat if it collects puss.

Go for it man. I'll give you my white coat so you can maybe have double the ED puss. Our select cohort of puss, I swear, literally eats away the speculum. It's like an episode of Stranger Things. The "upside down" is in there.


Sent from my iPhone using Tapatalk

 

[Birdstrike]

"Pus" has one "s".

At no time do I want to sit down in a chair and lose everything that I put in those pockets.

Oh, and fomites.

Maybe he meant puss. Which to me means something different. Maybe he's trying to collect it with his white coat.
Bird please clarify.

No excuses. 100% guilty of a,
Word C R I M E . . . https://www.youtube.com/embed/8Gv0H-vPoDc




(Random & unnecessary PS: See Weird Al in concert. Did a few days ago and it was hilarious & exceeded my expectations, by far. Bucket list stuff. Guy's a genius. #NerdPorn)

 

[deleted109597]

Somebody cares. Not you, but the guys paying for the surveys.

Yeah, but now those guys have 32 empty shifts for October, and I'm not working any of them.
Unless they pay me. A lot.

 

[Birdstrike]

Tell me more about this individually wrapped bacon.

Ever elusive my friend. Dream on. Only exists in ancient hymns and sung about by the locals (in Canada), like cold fusion, the fountain of youth and The Unifying Theory of all things:

Click here and forward to 01:35

 

[Birdstrike]

In anatomy lab in medical school someone put a puss in someone else's labcoat pocket.

This is how we ended up pregnant with Betty June.

 

[BAM!]

Seems like no one cares about having good scores. Has anyone ever seen a doc get fired over having scores which are too low?

 

[GeneralVeers]

Seems like no one cares about having good scores. Has anyone ever seen a doc get fired over having scores which are too low?

Not yet, though potentially it could happen. In my part-time world they simply don't put you on the schedule anymore. Official "firing" is bad for everyone

 

[BAM!]

Not yet, though potentially it could happen. In my part-time world they simply don't put you on the schedule anymore. Official "firing" is bad for everyone

 

[BAM!]

Not yet, though potentially it could happen. In my part-time world they simply don't put you on the schedule anymore. Official "firing" is bad for everyone

I know of it happening. Would there be any options for recourse, given ACEP statements and lack of statistical significance?

 

[deleted109597]

Be tough to find a lawyer. I'm not sure what you would gain from it. It's not like you'd want to go back to work there. And they wouldn't be paying you unless you couldn't get another job, which you clearly can.

 

[deuist]

I got a report back that had a whopping 2 surveys. 1 was a 5, the other was a 1. Guess how little I cared about what it, or any other press ganey has ever said?

I've been occasionally moonlighting at a civilian facility for the previous 2 years. Over the last 12 months I treated over 900 patients---not one of them has ever returned a patient satisfaction survey.

 

[gman33]

Pg is bs.
We all know it.

I just got my data for the last 6 months.
99% 1 month.
1% next month.

Same guy. Probably same care and pt interaction.

When i look at some of the comments, it seems like my bad scores are from people with unrealistic expectations. Things like expecting a specialist to come in and look at their chronic issue because they couldn't get an appointment in the office.

The problem is that my incentive pay is tied into this stuff.

From the hospital business standpoint, they'd probably want me to make the consultants come see the patient, order the extra MRI, or whatever else.

Patient, customer, would be happy.
More revenue for the hospital.

But there are just times when the demands are so crazy I just can't do it.

Should I just forget about this stuff and just assume I won't get that bonus money?

 

[BAM!]

Should I just forget about this stuff and just assume I won't get that bonus money?

depends on how much it is. i know a lot of people at my shop have that mentality, where it's only ~3-4k extra per year in incentives.

 

[GeneralVeers]

But there are just times when the demands are so crazy I just can't do it.

Should I just forget about this stuff and just assume I won't get that bonus money?

I had the same issue. One month in the top and the next month in the bottom. Since everyone generally scores > 75%, we are rewarding or punishing people based on random variation in the survey. That is why any monetary component tied to scores should be a token. Typically $500 per quarter is reasonable.

 

[Janders]

I don't think $ should be tied to individual docs unless you send out enough surveys to actually be statistically significant in differentiating between docs. Which is almost UNHEARD of.

Now, I'm not saying we can completely ignore these surveys. Patient care comes first, but there are obvious business and administrative relationships that benefit if you try to improve the press gainey...

IMHO, the key is to partner with administration, look at the PG surveys, and make realistic plans on how they can be improved. Dive into the details. Examples:
(1) The waiting room is dirty and the TV is too loud (easy to fix, get admin on that)
(2) I waited too long on radiology results
(3) I waited too long on lab results
(4) the front desk staff was rude
(5) You had no orthopedist available and I had to be transferred

Anyway, assuming your doctors aren't purposely antagonizing your patients, you can dig through these and find projects you can work on with hospital admin to not only improve PG scores, but improve your shifts...

 

[GeneralVeers]

I don't think $ should be tied to individual docs unless you send out enough surveys to actually be statistically significant in differentiating between docs. Which is almost UNHEARD of.

Now, I'm not saying we can completely ignore these surveys. Patient care comes first, but there are obvious business and administrative relationships that benefit if you try to improve the press gainey...

IMHO, the key is to partner with administration, look at the PG surveys, and make realistic plans on how they can be improved. Dive into the details. Examples:
(1) The waiting room is dirty and the TV is too loud (easy to fix, get admin on that)
(2) I waited too long on radiology results
(3) I waited too long on lab results
(4) the front desk staff was rude
(5) You had no orthopedist available and I had to be transferred

Anyway, assuming your doctors aren't purposely antagonizing your patients, you can dig through these and find projects you can work on with hospital admin to not only improve PG scores, but improve your shifts...

Administration doesn't care about any of those things. The administrators receive bonuses on metrics based on patient satisfaction. They are HUGE bonuses, some in fact bigger than your salary for the year. If that bonus is threatened, they will blame the doctors. No amount of reasoning, negotiating, or compromise will fix it. They will tell your group to improve or you're gone. Hence your group will have no choice but to implement some sort of measure tied to salary.