It was napkin math. So you aren't, most likely, investing most of the money you make in differential between the two jobs. For most, investments in the first three years are only going to be a minority of your income, likely as little as 25%. If you want me to be super generous, I'll factor that in, but I'll also extend the working career to 35 years, which assumes starting attending life at 32 and retiring at 67. In fellowship, most of the time you're going to average around 70k, sans moonlighting, which most fellows do but I'll ignore (I was making around 120k in fellowship, for example, with light moonlighting). But let's subtract the 70k average from the 272k average in IM. That gives us 202k, and we'll invest 25% of that because we're assuming you're a super financially responsible early career physician. That gives you 454.5k in cash advantage and 151.5k in investments. We'll let those investments grow at 10%. At age 35, when most specialists are starting out, you'll have $175,817 in invested retirement funds, for a total advantage of $630,317. Divide that out by 35 years and you get a total of $18,009 advantage added on to your salary for the 3 years you save.
If you project things out further though, you have to account for an investment rate of 25% for both IM and our theoretical specialist, so ultimately it just kind of becomes pointless to have factored in investments to begin with, as our theoretical internist ends up with a retirement savings of $20,039,729 while our theoretical cardiologist that lost 3 years of compounding but saves far more annually at the same 25% rate ends up with a pot of $29,416,500 (and more if they were moonlighting or even just saving some of their base salary). Even if we subtract the advantage factor of cutting out that 70k to even things out a bit, the internist still ends up short at $21,515,547. This is in addition to the cardiologist earning a lifetime earnings of $16,224,000 while the internist had lifetime earnings of $9,520,000. This leads to a net lifetime gain of $14,604,953 for cardiology in a worst case scenario in which cards only earns 70k a year in fellowship while saving nothing those 3 years and IM saves and invests 25% of their full salary from the start. So yeah, I guess the question for anyone who thinks it is a financially wise decision to do IM vs a specialty is "are you willing to sacrifice $5,000,000 per year for those early years of attending life," because that's what it amounts to. That's an advantage of $465,405 per working year gained over 32 years through the power of increased wages and compounding.
The only case where the math gets a bit messier is if you're starting medical school far later than average and also plan to start saving far more than average, then there are some scenarios that are very unique in which it could pay off financially. Then there's the aspect of whether one wants to be a cardiologist, and, well, you couldn't pay me a million dollars a year to do it, personally.