Are Metrics Killing People?

[WilcoWorld]

In this paper, the authors found that aggressive reduction of door-to-balloon times for presumed STEMI was associated with about twice as many false positive STEMI activations and a quadrupling of the in-house mortality rate in the FP-STEMI patients. The mortality of true STEMI patients trended down, but, unlike the other findings, this did not reach statistical significance.

We all gripe that metrics are figuratively "killing" docs and the practice of medicine, but this is (yet another) study to suggest that metric-driven care may be literally killing patients.

---

Members don't see this ad.

 

[Birdstrike]

To answer your question: "Yes." I've seen it and I have no doubt. Metrics-obsessed care kills people.

 

[Janders]

Yes.

We know now some metrics were bad ideas, both directly and due to "unexpected" effects of chasing them.

Beta-blockers for Acute MI are the former (yes, they are good in some cases, but they increase mortality in others)
Abx within 4 hours for pneumonia is the later (yes, likely helps the pneumonia patient a very little bit, but to succeed in this you know to shotgun antibiotics at so many people that don't need them... likely causing more harm than the good you are doing)

I would posit there is a 3rd type of harm, that focusing on a useless but harmless metric takes our energy away from others. For example, they have a fit if anyone who has a stroke doesn't have an NIHSS documented in their ED note. Now, if they have a hyper-acute stroke where you are considering TPA, NIHSS might just be a good idea. But a vague syncope case who had a 3-day-old lacune? What is the point of filling out an NIHSS on this patient? What harm comes of them if you don't? How annoying is it for the docs and nurses to be told they "messed up" the metrics? What type of system-energy do you need to expend to ensure 100% compliance on this metric?

So yes, at least 3 types of harm-- direct due actual bad medicine, direct due to bad metric implementation, and indirect due to waste of time/resources/attention.

I would further suggest the new sepsis metric fits all 3 of my harm types.

Now, all that venting out having been done-- I DO think metrics have some use. Metrics encourages the development of 24/7 rapid-access cardiac cath centers, which DOES help the care of STEMIs. I have seen metrics encourage hospitals to spend money on areas of care that are neglected. But a metric needs to be VERY precisely crafted, and the abstraction of the data needs to be easy AND appropriate... I believe most of our current metrics fail on these counts.

 

[Janders]

Oh another one that currently annoys me... Door-to-analgesia for long bone fractures.

Seems straight forward... if someone breaks a long bone, you should get analgesia in them within 50 minutes (I forget the exact time, its changed...).
If they NEVER get analgesia, they aren't counted as part of the metric.

However, if the patient DECLINES analgesia... even if they decline REPEATEDLY and this is documented in the RN and MD notes... but 3 hours into their ED course ask for and receive IV morphine, you FAIL the metric. There is no loop hole for patient refusal.

Irritating as hell, I feel like I'm constantly twisting little old ladies arms to make sure they don't just want a little touch of morphine when I meet them on the EMS stretcher, knowing that in 3 hours they'll want it and I'll have another fall-out...

 

[Apollyon]

Just as a note - nurse manager today tells me a code save we had got a P-G. He/the family gave us all 4s (but good comments). The guy was a defibrillated, hypothermatized, CPRd code save! 4s, indeed!

 

[Janders]

Just as a note - nurse manager today tells me a code save we had got a P-G. He/the family gave us all 4s (but good comments). The guy was a defibrillated, hypothermatized, CPRd code save! 4s, indeed!

LOLZ.
Don't you ever feel like you're working inside Catch-22 or some Vonnegut novel?

 

[Arcan57]

Oh another one that currently annoys me... Door-to-analgesia for long bone fractures.

Seems straight forward... if someone breaks a long bone, you should get analgesia in them within 50 minutes (I forget the exact time, its changed...).
If they NEVER get analgesia, they aren't counted as part of the metric.

However, if the patient DECLINES analgesia... even if they decline REPEATEDLY and this is documented in the RN and MD notes... but 3 hours into their ED course ask for and receive IV morphine, you FAIL the metric. There is no loop hole for patient refusal.

Irritating as hell, I feel like I'm constantly twisting little old ladies arms to make sure they don't just want a little touch of morphine when I meet them on the EMS stretcher, knowing that in 3 hours they'll want it and I'll have another fall-out...

I just had to present this core measure to our system leadership and documented patient refusal drops them out of the population. If your hospital is abstracting differently, it's due to faulty interpretation of the measure not the measure itself. Additionally, giving a patient an oral pain medicine first (18 or older) or documented pre-hospital analgesia (either EMS or pt provided, all ages) will cause them not to be included in the study measure. Other (although not all-inclusive) exclusions include age <2, dying in the ED, and documented contraindication to pain meds.

 

[WilcoWorld]

LOLZ.
Don't you ever feel like you're working inside Catch-22 or some Vonnegut novel?

Only when I think about it...

 

[Janders]

I just had to present this core measure to our system leadership and documented patient refusal drops them out of the population. If your hospital is abstracting differently, it's due to faulty interpretation of the measure not the measure itself. Additionally, giving a patient an oral pain medicine first (18 or older) or documented pre-hospital analgesia (either EMS or pt provided, all ages) will cause them not to be included in the study measure. Other (although not all-inclusive) exclusions include age <2, dying in the ED, and documented contraindication to pain meds.

Arcan, can you link me to firm proof of this by any chance?

I haven't been able to find an exclusion for patient initial refusal followed by desire for later analgesia on my review of the flowchart. I also don't see a written EMS exclusion (one exists for door-to-EKG time). Downloading the metric PDF from qualitynet.org shows limited exceptions of:
"Excluded Populations:
• Patients less than 2 years of age
• Patients who expired
• Patients who left the emergency department against medical advice or discontinued care "

J

 

[Arcan57]

Arcan, can you link me to firm proof of this by any chance?

I haven't been able to find an exclusion for patient initial refusal followed by desire for later analgesia on my review of the flowchart. I also don't see a written EMS exclusion (one exists for door-to-EKG time). Downloading the metric PDF from qualitynet.org shows limited exceptions of:
"Excluded Populations:
• Patients less than 2 years of age
• Patients who expired
• Patients who left the emergency department against medical advice or discontinued care "

J

I'll try and find the commentary.
http://www.qualityreportingcenter.c..._Abstraction-Tips-and-Tricks-AM-508-FINAL.pdf

Question and answer 2.

 

[Janders]

Freaking brilliant!

Of course, they did NOT add this clarification to the new version of the rules (which come out multiple times a year) OR in the release notes, but instead in a non-indexed Q&A file. Jebus. Needless to say, my abstractors have not seen this. Le sigh.

Owe you a beer.

 

[LaBusqueda]

I really wish there was a real public service/ad campaign on this crap!
Our professional organizations (all of them) have proven impotent on these things and simply pander down over and over. Is it to save face somehow?

It's time for docs to take this to the people and hopefully get people on our side for once!

Like the pay for utilization with CT and such mentioned in the other thread, while not a metric as stated here, I cannot imagine this going over well with the population if the media took a good spin with it. Would love to see the responses to future potential pieces such as "docs getting paid to deny you tests" or "New performance metrics incentivize doctors to deny care", etc.

Maybe start it grass roots style and go with YouTube vids as I am sure there would be no help from the powers that be. Who would be in for that?

I am jaded enough that I would get on board with this and speak out for sure.

 

[deleted109597]

If your hospital is abstracting differently, it's due to faulty interpretation of the measure not the measure itself.

Shocking. Similar to all of the bull**** TJC rules about drinks (which aren't actually a rule at all).

 

[Tiger26]

Shocking. Similar to all of the bull**** TJC rules about drinks (which aren't actually a rule at all).

Yeah, not CMS or JCAHO or anything medically related. Only OSHA says beverages can't be in a 'patient care area'--if somebody codes near the computers where I'm drinking my coffee, our plan is to drag them into a treatment room, so we now just disregard the nurse administrators on this rule

 

[xaelia]

In this paper, the authors found that aggressive reduction of door-to-balloon times for presumed STEMI was associated with about twice as many false positive STEMI activations and a quadrupling of the in-house mortality rate in the FP-STEMI patients. The mortality of true STEMI patients trended down, but, unlike the other findings, this did not reach statistical significance.

We all gripe that metrics are figuratively "killing" docs and the practice of medicine, but this is (yet another) study to suggest that metric-driven care may be literally killing patients.

Was anyone able to get the full text of the Mayo Clinic proceedings DtB QI paper cited?

Would like to blog about it, but I don't discuss things I can't read full text ... and our Library couldn't seem to get at it yet.

 

[WilcoWorld]

Was anyone able to get the full text of the Mayo Clinic proceedings DtB QI paper cited?

Would like to blog about it, but I don't discuss things I can't read full text ... and our Library couldn't seem to get at it yet.

I have only read the abstract so far. Will be back at work Sunday. If I can get full text then I'll send it your way.

 

[Old_Mil]

The only way to win is not to play.

 

[RustedFox]

The only way to win is not to play.

Greetings, Professor Falken.

A strange game. The only winning move is not to play.

How about a nice game of chess ?

 

[Arcan57]

Freaking brilliant!

Of course, they did NOT add this clarification to the new version of the rules (which come out multiple times a year) OR in the release notes, but instead in a non-indexed Q&A file. Jebus. Needless to say, my abstractors have not seen this. Le sigh.

Owe you a beer.

Glad to help a brother out. Honestly, the difference in middle and top quartile on most metrics is appropriately excluding patients that fell out but actually should never have even been in the population. In a similar vein, our PG percentile just jumped 30 points because they accidently surveyed some patients that had been screened out and we were able to get those surveys thrown out. In a disturbing sort of way, it's fascinating how common errors in abstraction occur.

 

[Janders]

Completely agree. We have had prior issues with abstraction (counting cases that were clearly outliers, amongst other details) which is why I've now read the full text of all the ED measures, and make it a point to audit any poorly performing metrics down to the case-by-case full chart level.

As far as this metric (analgesia for long bone frx), our actual main issue was pedi patients coming in with relatively minor fracture (green stick forearm, etc) and sitting in the waiting room comfortably for 40 minutes with a bag of ice. By they time they get back, Xray is done but often we order some oral motrin... it is given at the 65 minute mark and the metric is failed.

Gotta either get them back within 20 minutes (my preference) or make it possible for the triage-both RN to dispense weight-based APAP/Ibuprofen (change the triage RN protocols, more importantly find a way to stock the med out there so they don't have to walk back into the ED to the pixis 10x a shift...).