Temple sounds out of the question, seems like you'd be miserable there. Essentially it seems like a question of whether you should withdraw from VCU or UM. Both schools are solidly mid tier so I would personally not base this decision on prestige - if you work hard, both should get you to where you want. Instead, do some introspection and decide which is more important to your happiness - staying close to your family, p/f grading, and tiny debt vs. new location/weather and better fit with school. Taking extra loans to go to the school you are happier at is always justifiable.
Also include cost of living in your estimates, because they are all going to be vastly different. I'm guessing UM will be closer to 55k than 35k if you did. If Brown is not much more than 100k more in total debt than UM will be, then it is clearly the top choice, especially for the unranked p/f and prestige factor in case you have any inkling of pursuing ortho or ENT in the future.
PS Miami also has a 150 kid class size so I'm not sure what difference this makes since large class size is included in your VCU/Temple cons