I think that it doesn't matter what other people think! 🙂
$120K is a lot of money, but you are going to be a doctor and will be able to pay it back, with interest. So, it's entirely your call whether or not the advantages Jefferson offers you are worth it. It's your money, your life, and your call.
I'm not sure why anyone would allow anonymous strangers on the internet to sway them one way or the other with such big decisions. In fact, most don't. They solicit opinions and then follow the ones that confirm what they wanted to do all along.
Since you clearly would really rather be in Philly, I think you should go with your gut, rather than risk being unhappy in Detroit for $120K that is going to be a drop in the bucket in relation to your lifetime earnings. There -- now you have a post confirming that what you want to do anyway is perfectly okay. 🙂
Honestly, it is! You don't need permission from me or anyone else. You also don't need anyone guilting you into taking the money and doing something you really don't want to do.