It matters in the sense that the more debt you have the more your income is going to take a hit in those 10 years. The last thing you want is to work for the next 7 years of your life toward a goal and then not be able to live a somewhat lucrative life in the next 10.
For primary care that really only matters depending on where you practice. If you're a primary care doc making $150k a year in rural Tennessee it's a lot different than if you're practicing in inner-city Boston. In rural Tennessee you're very comfortable and, truth be told, your salary is probably higher than if you were in Boston as well. (Debatable but typically rural docs make more net salary because the rural hospitals have to pay more to keep people). The link I posted above sort of illustrates that. In that study they found that, in some cases, the inner-city Boston doc was actually coming out in negative incomes.
Again, you can't bet on income based repayment being there 4 years from now. Not at all. We had a financial advisor presentation a week or so ago and that's exactly what he told us. It's there now but we shouldn't make any financial decisions planning on it to be there in 3 years when we're done.
To a certain degree med students can just not give a damn. So many med students try to live frugally, never eating out, etc. while they're in school and across four years might save about $15k. In the grand scheme of things that's really not that big of a deal. However, if you're talking $100k, it does deserve a longer look.
I don't envy your situation and the decision you have to make!
[Edit]: And, I should say, I don't know any primary care docs in rural TN making $150k. Most are making at least $200 and up. Somewhere there are family practice docs struggling to get by but in my experiences they're not in rural America.