As a Yale med student, you will spend most of your time on campus at the med school, or possibly on central campus too if there is something you have to do. You'll frequent some of the off-campus housing areas, the local bars, restaurants, clubs, and shops, but the other parts of New Haven you won't be visiting. As for safety, New Haven is much safer than a larger city like Baltimore as long as you stay in the right parts (which will have all the places that you would consider going to anyways). It won't have what a big city offers, but if you are craving that, you can just catch an hour 40 minute long Metro North train from Union station to Grand Central, NYC.