If I were you, I'd drop UCSD since they are not strong in your research area, and attend the others since it sounds like they all are. Chapel Hill is *not* the sticks.
It may seem like a lot of work and effort to attend four interviews after you're already accepted to an admittedly awesome program, but it's worth it just so you don't wind up asking yourself "what if" for the next eight years. Two years ago when I applied, I wound up attending 17 interviews, and the school I ended up at was one of the last ones I visited (#15). Now I will readily admit that I'm more OCD than most (
). That being said, I was invited on two other interviews, and the one single thing I regret about my app year was not attending one of those two declined invites. I had an opportunity to visit that school last summer, and it's fabulous. I don't know if I would have chosen that school over the one I'm at now, but the point is that I never even gave myself the opportunity to see if I liked it better, and that's what I regret.
Basically, you have no way to judge one school or city against another unless you go there and see them for yourself. You talk about not wanting to be in a small town, but nice as it is for a college town, Ann Arbor isn't exactly a city either! If you think there is any chance you might want to go some school, you owe it to yourself to make the most informed decision you can. Go there, see what it's like, and if you don't like it better than U Mich, you can always withdraw. Obviously, you have to be realistic in terms of time and money, but you didn't mention either time or money as being the limiting factor in your decision to drop some interviews. I can completely sympathize if that is the issue here; if you can't afford to attend all the interviews, you'll have to cut some. But don't give up invites to schools you might realistically want to attend just because it's such a pain to go on so many trips and you want to just get the process over and done with.