This.
Last year at my med school, two students were robbed at gunpoint in broad daylight (7:30 a.m.) ON CAMPUS. This year, a student was physically assaulted right across the street from campus and again it was in broad daylight. Muggings happen anywhere and at any time and anyone who thinks they don't hasn't lived in these rough neighborhoods.
OP, I would talk to your clinical director. If you're not comfortable, they need to know. If you really think there's a risk, you shouldn't be expected to take that risk for a FM rotation.
I don't. That same activity was going on around my med school. Shootings, drug deals, prostitution. It wasn't all that uncommon to see police arresting prostitutes right across the street. We were on lock-down once last year because of a shooting not involving the school, but down the street. I'm on rotations now, but the students who are there this year were on lock-down a few months ago because of a bomb in the student parking lot. Students had to stay in the lecture hall or lab while the bomb squad was called. There are some places that just aren't safe and unless we're there, we can't tell the OP to just suck it up.