Wrt Yale cons:
(1) You don't have to take a 5th year, and choosing not to do so won't put you behind your classmates. If you want to do a competitive surgical specialty/subspecialty, it will benefit you to take a year regardless (but again, you don't have to).
(2) There are in-person classes for the master courses, you just don't have to go to them and each course only has 1 required, summative exam.
(3) You think having too relaxed of a structure won't work for you because you're not used to it. I get it, I had the same reservations. But what you'll find is that most medical students end up primarily learning via 3rd party resources on their own time anyway, and you'll appreciate not having to go to class or have tons of tests/assignments when you're studying for standardized exams. And the truth of the matter is that if you were driven enough to do this entire process and got into multiple Ivy League schools, structure is not something you'll need to worry about. Plus you won't always have structure. Adults have to manage their time and figure out ways to get things done. I don't know you but I'm confident you can do that.
(4) Can't really say much about campus, agree that it isn't really nice lol. But not sure that's a *real* con.
The mental benefits of having P/F clerkship just cannot be understated. And for every medical school, community work is largely a self-driven activity. At a place like Yale, all of the extra free time allows you to become very involved in the community and make a difference without having to worry about failing tests.
It sounds like you have a bit of a preference for Columbia and it seems mostly based on vibes, but your Yale "cons" are not really cons, whereas your Columbia cons absolutely are.
I vote Yale, but you'll be fine either way. Good luck!