All Branch Topic (ABT) Leapfrog Results

[militaryPHYS]

Interested to hear thoughts on leapfrog and its application within the MHS

Sixteen Military Hospitals Receive Leapfrog's Highest Safety Grade - DHA.mil

A hospital must meet specific criteria to be eligible for participation in the Leapfrog safety assessment.

dha.mil

---

Members don't see this ad.

 

[DrMetal]

Interested to hear thoughts on leapfrog and its application within the MHS

Sixteen Military Hospitals Receive Leapfrog's Highest Safety Grade - DHA.mil

A hospital must meet specific criteria to be eligible for participation in the Leapfrog safety assessment.

dha.mil

It's not hard to be "safe" when you have low volume and low acuity!

 

[militaryPHYS]

Actually volume is a specific area they focus on. Numbers are reported publicly on leapfrog’s website for each institution. It affects overall score but not sure how it is weighted. I am trying to dig deeper into it

 

[DrMetal]

I've had either first hand experiences (or know of good second-hand reports) of all that Navy MTFs. They're not doing much. Most complex care is getting deferred out.

This is like me bragging about shooting a 78 at my local golf course, not counting all my penalty strokes and taking a dozen mulligans.

 

[militaryPHYS]

Are you saying there is no validity to Leapfrog results?

 

[militaryPHYS]

Leapfrog Hospital Safety Grade and Surgical Volume

The Leapfrog Hospital Safety Grade assigns hospitals a letter grade (A through F) based on 27-30 measures of patient safety, drawn from the Leapfrog Hospital Survey, CMS data, and other sources. Surgical volume is not a standalone measure in the Safety Grade but is indirectly incorporated through related safety and outcome metrics.

Indirect Role of Surgical Volume:
• The Safety Grade includes outcome measures like postoperative complications, surgical site infections, death from serious treatable complications (PSI 4), and accidental punctures or lacerations (PSI 15), which can be influenced by surgical volume.
• Hospitals with low volumes for high-risk procedures may have higher rates of adverse events due to limited experience, which could negatively impact these outcome measures.
• Additionally, the Safety Grade includes process measures like safe surgery practices (e.g., adherence to surgical checklists), which are often stronger in high-volume centers with established protocols.

Weighting in the Safety Grade:
• Each of the 27-30 measures in the Safety Grade is assigned a weight based on its evidence base, opportunity for improvement, and impact on patient safety. The exact weights are determined by an expert panel and updated periodically.
• Surgical volume is not explicitly weighted as a standalone measure, but its influence is embedded in the outcome and process measures mentioned above. For example:
• Outcome measures (e.g., PSI 4, postoperative sepsis) typically account for ~50-60% of the total Safety Grade score, with individual measures weighted between 2-5% each.
• Process measures (e.g., safe surgery practices) account for ~40-50% of the score, with similar per-measure weights.
• If a hospital’s low surgical volume leads to poor performance on outcome measures (e.g., higher mortality or infection rates), this could reduce its Safety Grade, but the effect is diluted across the many measures considered.
• The Leapfrog methodology emphasizes standardized, risk-adjusted metrics, so volume-related impacts are not isolated but rather reflected in broader safety performance.

Example:
• A hospital performing only 5 pancreatic resections annually (below Leapfrog’s threshold of 16) might have higher complication rates, contributing to a worse score on outcome measures like postoperative sepsis or mortality. This could lower its Safety Grade, but the impact would be spread across multiple weighted measures rather than tied directly to the low volume.

 

[DrMetal]

Example:
• A hospital performing only 5 pancreatic resections annually (below Leapfrog’s threshold of 16) might have higher complication rates, contributing to a worse score on outcome measures like postoperative sepsis or mortality. This could lower its Safety Grade, but the impact would be spread across multiple weighted measures rather than tied directly to the low volume.

I don't care how you fudge or weigh the numbers.

A low N is a low N, no matter how you slice it. You shouldn't be drawing any conclusions from it (positive or negative).

It makes no sense, that in their own explanation, they admit a low N, but somehow find a work-around.

That's like me shooting a 76 on a golf course, once, despite all of my other scores easily being in the 90s . . . and then I brag about being a single-digit handicap. (I'm all about the golf analogies)

 

[Dr. Wexler]

DHA loves to collect merit badges.

Reminds me of Walter Reed Bethesda claiming to be a “trauma center” because they met some ACS criteria. Nevermind that🙁1)”trauma center” is actually a geopolitical designation and Maryland does not recognize them as a trauma center
(2) they don’t function as a trauma center anymore than a rehab hospital serves as a trauma center

I know little of Leapfrog but suspect similar forces and gamesmanship are at work. Good bullet points for your yearly report card.

 

[DrMetal]

DHA loves to collect merit badges.

Reminds me of Walter Reed Bethesda claiming to be a “trauma center” because they met some ACS criteria. Nevermind that🙁1)”trauma center” is actually a geopolitical designation and Maryland does not recognize them as a trauma center
(2) they don’t function as a trauma center anymore than a rehab hospital serves as a trauma center

I know little of Leapfrog but suspect similar forces and gamesmanship are at work. Good bullet points for your yearly report card.

Yeah, same with Camp Lejern , claiming to be a "Level 3" Trauma Center . . . meaning they can repair scalp lacerations, that I used to do on a ship all the time. My USS ship was definitely a trauma (more like, drama) center.

 

[Apollyon]

Yeah, same with Camp Lejern , claiming to be a "Level 3" Trauma Center . . . meaning they can repair scalp lacerations, that I used to do on a ship all the time. My USS ship was definitely a trauma (more like, drama) center.

Was "I'm going to jump off the fantail" a real thing?

 

[afteranesthesia]

Leapfrog Hospital Safety Grade and Surgical Volume

The Leapfrog Hospital Safety Grade assigns hospitals a letter grade (A through F) based on 27-30 measures of patient safety, drawn from the Leapfrog Hospital Survey, CMS data, and other sources. Surgical volume is not a standalone measure in the Safety Grade but is indirectly incorporated through related safety and outcome metrics.

Indirect Role of Surgical Volume:
• The Safety Grade includes outcome measures like postoperative complications, surgical site infections, death from serious treatable complications (PSI 4), and accidental punctures or lacerations (PSI 15), which can be influenced by surgical volume.
• Hospitals with low volumes for high-risk procedures may have higher rates of adverse events due to limited experience, which could negatively impact these outcome measures.
• Additionally, the Safety Grade includes process measures like safe surgery practices (e.g., adherence to surgical checklists), which are often stronger in high-volume centers with established protocols.

Weighting in the Safety Grade:
• Each of the 27-30 measures in the Safety Grade is assigned a weight based on its evidence base, opportunity for improvement, and impact on patient safety. The exact weights are determined by an expert panel and updated periodically.
• Surgical volume is not explicitly weighted as a standalone measure, but its influence is embedded in the outcome and process measures mentioned above. For example:
• Outcome measures (e.g., PSI 4, postoperative sepsis) typically account for ~50-60% of the total Safety Grade score, with individual measures weighted between 2-5% each.
• Process measures (e.g., safe surgery practices) account for ~40-50% of the score, with similar per-measure weights.
• If a hospital’s low surgical volume leads to poor performance on outcome measures (e.g., higher mortality or infection rates), this could reduce its Safety Grade, but the effect is diluted across the many measures considered.
• The Leapfrog methodology emphasizes standardized, risk-adjusted metrics, so volume-related impacts are not isolated but rather reflected in broader safety performance.

Example:
• A hospital performing only 5 pancreatic resections annually (below Leapfrog’s threshold of 16) might have higher complication rates, contributing to a worse score on outcome measures like postoperative sepsis or mortality. This could lower its Safety Grade, but the impact would be spread across multiple weighted measures rather than tied directly to the low volume.

Have you drank so much koolaid you are trying to argue against the known issue of volume with Leapfrog results? I remember all the ridiculous certificates and merit badges the AF institution I used to work at gathered....it was so dumb. Half those nurses couldn't function in an actual hospital. Please don't debate the known issue of volume by asking us if the Leapfrog results mean anything. Do you chase ER docs asking about their Press Ganey scores too (The Pit)?

 

[militaryPHYS]

NDAA17 mandated that DHA utilize civilian outcome data metrics so we can be evaluated against our peers. Volume is a known issue in the MHS and I wanted to discuss others thoughts on how it impacts the overall score.

Lots of very well respected civilian hospitals with tons of volume in our area did not score an A. These are outcomes based measures.

 

[pgg]

Interested to hear thoughts on leapfrog and its application within the MHS

Sixteen Military Hospitals Receive Leapfrog's Highest Safety Grade - DHA.mil

A hospital must meet specific criteria to be eligible for participation in the Leapfrog safety assessment.

dha.mil

It's good news - obviously beats the alternative. There are structural and management issues with the MHS but there's no reason to dismiss this as meaningless or irrelevant. The military does do some things well, and I'm not surprised that measurable safety metrics are among them. Teaching to the test and hitting wickets are among our core competencies and I'm sure our stakeholders display great synergy skating to where that puck is.


I just looked up the hospital system where I work. The main hospital is a 500ish bed facility. It got a grade of A. I think we do a pretty good job in a broad sense, despite some issues with one of our surgical subspecialty departments. One criteria (wound dehiscence after surgery) had a score of 4 per 1000 which was the "worst" of any hospital ... can't really say that particular problem was on my radar here. Apparently we're also very bad at "accidental cuts and tears" ... and not so good at "air or gas bubble in the blood" (6 per 100K patients) ... we got 100 on the handwashing bit though.

One of our affiliated community hospitals got a B. The criteria that were in the red for that place weren't really on my radar as deficiencies or problems, but who knows. I don't spend much time there.

There's a smaller community hospital nearby that got an F. I don't work there but as far as I know it doesn't have a reputation for being a house of death and terror, despite the failing grade. One of its metrics is even the WORST of any hospital that Leapfrog grades.


Anyway. I didn't read too deeply on their methodology, but I'm sure it's EMR data mining and surveys. I'll just say that comparing hospital systems using different EMRs (and they're all different, even the ones using the "same" EMR) is a process fraught with sources of error. Broadly speaking my impression of organizations like this is that a former sweater-wearing, clipboard-carrying, Nurse Corps officer would fight right in.


But an A is an A.

 

[DrMetal]

I'm sure our stakeholders display great synergy skating to where that puck is.

But an A is an A.

Still, it's a little hard to brag about getting an 'A' on a 5-question test.

Get an 'A' on a 500-question test (a la your current hospital), now that's more meaningful.

 

[militaryPHYS]

View attachment 403388



Still, it's a little hard to brag about getting an 'A' on a 5-question test.

Get an 'A' on a 500-question test (a la your current hospital), now that's more meaningful.

It’s the same process at every hospital.

Let’s try this: Which independent, civilian grading system would you be satisfied with as a way to compare ourselves and constantly be improving?

 

[HighPriest]

This is always going to be contentious because I think almost everyone feels that we should have clear metrics by which to evaluate our standards and by which to guide improvements, but all of the ones of which I am aware are fraught with problems with regards to how they measure, how they weight, and whether the things they’re measuring even matter in many cases. The ideal tool would be retrospective, prospective, and introspective. Meaning they would data mine (like they all do), but also follow a cohort of patients within the system to compare their long term health to a similar cohort at other institutions, and the tool would use that data to determine whether or not the metrics we’re using make any difference at all. (One example of what I mean: our institutions is surveyed by DNV and one of the metrics they use for OR performance is whether I document a heart and lung exam in the preop holding area. Which is ridiculous because I am absolutely terrible at heart and lung exams and I have an anesthesiologist who is probably pretty good at them also doing that same exam and documenting it in preop holding. So there’s zero chance that my documenting it as well has any benefit to patient care or safety).

A good rating is a good rating. I agree. And I definitely agree that if nothing else the military is good at making sure they meet metrics. It seemed to be all that mattered to them. It’s pretty highly regarded in my institution now, but nothing like the military where they would follow that light right off a cliff.

But a lot of these rating systems are like restaurants claiming to have the “best pasta in town” as rated by an online website based in Islamabad. It isn’t entirely clear what it even means. Or like when you get those fliers in the mail from institution “X” stating that you’ve been identified as “one of the best” physicians in your community and if you send in $60 they’ll make sure your name appears in their mailer indicating that…..

 

[militaryPHYS]

This is always going to be contentious because I think almost everyone feels that we should have clear metrics by which to evaluate our standards and by which to guide improvements, but all of the ones of which I am aware are fraught with problems with regards to how they measure, how they weight, and whether the things they’re measuring even matter in many cases. The ideal tool would be retrospective, prospective, and introspective. Meaning they would data mine (like they all do), but also follow a cohort of patients within the system to compare their long term health to a similar cohort at other institutions, and the tool would use that data to determine whether or not the metrics we’re using make any difference at all. (One example of what I mean: our institutions is surveyed by DNV and one of the metrics they use for OR performance is whether I document a heart and lung exam in the preop holding area. Which is ridiculous because I am absolutely terrible at heart and lung exams and I have an anesthesiologist who is probably pretty good at them also doing that same exam and documenting it in preop holding. So there’s zero chance that my documenting it as well has any benefit to patient care or safety).

A good rating is a good rating. I agree. And I definitely agree that if nothing else the military is good at making sure they meet metrics. It seemed to be all that mattered to them. It’s pretty highly regarded in my institution now, but nothing like the military where they would follow that light right off a cliff.

But a lot of these rating systems are like restaurants claiming to have the “best pasta in town” as rated by an online website based in Islamabad. It isn’t entirely clear what it even means. Or like when you get those fliers in the mail from institution “X” stating that you’ve been identified as “one of the best” physicians in your community and if you send in $60 they’ll make sure your name appears in their mailer indicating that…..

Solid points. Thanks for the dialogue.

Not having enough patient volume/complexity is very openly talked about at the highest levels. I think, at the very least, leapfrog helps us compare like-MTFs and identify areas for improvement at certain sites.

 

[DrMetal]

Solid points. Thanks for the dialogue.

Not having enough patient volume/complexity is very openly talked about at the highest levels.

They've been talking about it for years, no action has taken place. When the lobbies of our clinics are full again, and when surgeons start taking complex cases (instead of deferring them out the network, b/c they don't have adequate nursing or anes support, or because they just don't want the liability of it) . . . then I'll believe change has occurred. So far it has not, our MTFs are still ghost towns.