Disclaimer: I'm going to Yale starting this summer.
Warning: wall of text
I think that the main things that differ between these schools are the location and the educational environment/atmosphere/philosophy.
I had some worries about New Haven, but after talking with several students I feel better about it. Is it going to be a vibrant place with tons of stuff going on like a major city? Of course not. But it seems like there is a decent amount of activities and happenings around New Haven, the food seems good, and the Yale community is solid. Also, the demographics are suited for medical education because there is so much variation (in terms of SES, geographical, and race/ethnicity). However, if you’re a city person I won’t lie to you and say you’ll like it more than Philly, but it seems better than the rap it gets.
In terms of the educational environment, I know a few people at Penn and I think that it is pretty different from Yale. Penn just switched to preclinical P/F. While that’s obviously a good move, it says something that they were so slow to make the change compared to peer institutions. My friends who go to Penn are, to be perfectly honest, kinda hardos, and they like the environment because it stresses them out and there is more competitiveness there than peer institutions. That’s totally not for me. Is it for you? It’s so so different than Yale. This seems like the most acute difference.
One more thing: Penn has the better hospital system hands down. I was worried that while Yale-New Haven is very well regarded, it isn’t the same quality of the hospital systems of other schools I got into (esp. Michigan), and I didn’t know how that would affect me as a student. I talked with several students and a former professor/clinician, and they all reassured me that they felt the clinical education was top of the line and described some things about their rotations that made me feel a lot better.
Hope that helps! Feel free to PM for additional questions, etc.