I don't think that should be a blanket decision when deciding where to apply. First, as others have mentioned, you should consider neighborhoods prior to cities. There are many neighborhoods in Washington, D.C. that I get nervous just driving through, but most of the universities are located in posh neighborhoods in NW where I would not think twice about living or working (further, some of the formerly sketchy neighborhoods in SE are currently getting gentrified-- but that's a whole 'nother issue). Second, even if a university is in a high crime neighborhood, you're not required to live there. Temple, which as Ollie notes is located in a hotbed of crime (North Philly, perhaps the worst region of the city), is right on the Broad Street line, which would be easy to ride to and from a swanky and utterly safe apartment in Center City (or even catch a train to the suburbs, if you are THAT wimpy). Third, even within bad neighborhoods, universities tend to be safe. This is true of Temple-- I believe their campus is relatively safe and well policed. Also in Philly, Penn (which is in the part of Philadelphia where the Fresh Prince was born and raised) is surrounded by several well-policed and relatively safe blocks with nice restaurants and bars (but wander too many blocks west and you safety dissolves). Fourth, small college towns are not immune to crime. In fact, in my experience, campus in small college towns tend to attract local townie riff raff who have nothing better to do than to assault university students. Consequently, I have sometimes felt more sketched out in college towns than in New York City or other big, high crime areas.
I would not eliminate any universities in this country based on crime rates. You can always assess safety at interviews and turn down offers if you feel uncomfortable.