I remember the initial experiences at my school on interview day. I felt a real connection with some of the 1st year students. I was excited to see the faculty and recognized some of their names from my long hours on the internet ("I read his blog! Cool!"). I hit it off so well with my interviewer--one of the faculty professors--that I thought somehow I was entering a set of almost magical professional relationships. Though I remember that the campus seemed grubby… it was a bit bleak and I had to commute, but given the state school prices I was all in.
Then time happened. Many of the professors who had been introduced to us left. New ones took their place to teach us. My initial impressions of professors proved reliable only half the time. Some people who are great to speak with one-on-one are terrible educators. Some great professors can come off cold at first glance. And when someone seems a bit crazy, that can be a good thing or a bad thing.
Aside from occasionally seeing them at a bar, I don't have much to do with the students ahead of me, and I'm just fine with that. We're all busy with our own stuff.
One of the biggest surprises has been the way the campus has grown on me. I only see a small sliver of it, but I enjoy it and the way it fits into the terrain.. I've seen many sunrises and sunsets as I trudge to and from the parking lot. Sitting here on winter break after so many days of not being in school, I'm shocked by how much I cherish those images in my mind.
Love at first sight is an interesting thing. A brief sample of experience gets projected out in time, making it easy to imagine all the wonderful possibilities that will be missed if things fall apart. Like having some profound conversation with a stranger in a bar and then losing her number (or it was like that back before facebook, but perhaps I'm dating myself now). It probably wasn't going to become a lifetime romance. Sleep easy.
It sounds like you feel torn, even as you may be moving toward the solution with less debt (a move I would encourage). For what it's worth, no matter where you go there will be professors you love and professors you hate. There will be fellow students who seem like long lost friends and those who you dread having to work with. Even impressions regarding the campus and commute can change over time. It's strange.
Having an extra $40k to throw around after you graduate though, that gives you some options. Having lower payments per month means you might be able to take a better job in a town you want to live in as opposed to a patient mill in Nowhere, USA. You could maybe afford to do a residency and advance yourself professionally. Or with lower overhead it might be easier to take the risk and try to start your own practice.
You aren't giving up a magical experience at UM for the abstraction of having a smaller number sitting in the electronic records of some lender. You're choosing whatever is behind Door Number 1 over whatever is in the Mystery Box. And in return you get this new set of choices in a few years that could take you somewhere great.
tl;dr - My initial impressions of my school were wildly inaccurate, so caveat emptor. You'll appreciate having more spending money later.
