If you have the opportunity to graduate debt-free, take it. Your savings could be used to buy a house or invest. I hear what you are saying about finding your people in a small student body, but the experience of medical school is stressful enough that you'll find people no matter what.
It's not like Baylor is some backwater school. Take the scholarship and enjoy the doors that are opened for you just because you have no debts to pay off. Enjoy being at a P/F school with a solid reputation. This is a no-brainer.