I spent 2 years as a grad student at Temple Medicine, and it is in the worst part of the city. People will harass you, and I've been cursed at and spit at various times (I was just walking). Most of these people just want to remind you that you're in their woods, despite their woods being in the shadows of the medical school buildings.
All that being said, I only felt physically threatened a handful of times, and I was commuting at all hours of the night (3-4 AM was very common), so I was certainly taking some risks for the sake of my experiments. I'm also a sizable guy though, so I think that prevented some would-be harassers. I parked on the street, didn't keep anything valuable in my car, and never had a problem with break-ins, although a few of my friends did.
If you stand outside for any considerable amount of time, you'll likely see 1-2 uniformed officers within a few minutes, and there is normally 2 parked at the hospital practically 24/7. It is definitely not the safest, but Temple does a lot to keep it as safe as possible. Incidents do happen though, and I wouldn't feel comfortable travelling even a block off Broad Street at night.
