Obviously there's no answer to this question. You need to figure out what's most important to you. The factors to consider are: Curriculum/Learning Style, Financial, Lifestyle/Living Environment, Reputation, Future Goals.
I also went to an undergrad Ivy school, worked, and am now at a state med school. You're right, my undergrad experience was much different from my state school friends, but to be honest, I don't feel any smarter or dumber than the smart state school people. Smart people are smart people, dumb people are dumb people... Going to a school with smart people won't make you smart. And having people think you're smart won't make you smart either. The same goes if you're dumb.
What matters is what you get out of your education and how you translate that into results, performance, and outcome. Even the highly educated student can be a ****** in the hospital.
I will admit there are advantages: there's definitely a Wow Factor to graduating from a bigger name school. It opens doors to opportunties and people tend to give you the benefit of the doubt. At the same time, people will hold you to a higher standard that sometimes is not possible and will piss on you when you fall short of standards that someone else set up for you. The Wow Factor is short lived. At some point after doors have been opened for you, you have to prove that you deserve the Wow, which is really tough some times. The people who fall the hardest are the ones who actually believe they're entitled to the Wow and refuse to learn or take criticism.
Just as there are stereotypes of people going to big name schools, there're going to be sterotypes of people in small name schools, which is unfair at times when you don't get the respect that you deserve for all your hard work. I think people in this situation need to be strong, self-confident, but humble enough to maximize their learning opportunties. You have to know deep down that you put the time and effort and have all of that show in your attitude and performance. At that point, people will respect you regardless of what school you went to.
From here to eternity, people will judge you... whether it's by your educational background, ethnic background, socioeconomic status... whether your applying for med school, residency, fellowship, or a job after training. So it all depends on how you manage Perception and Reality. Some people will care, some people won't. No one kicks an AOA Harvard/Hopkins Step I 99% med student out of bed for eating crackers. And when you're patients see that you went to Stanford Med, they may be more relaxed before you cut their eye than they would be if they saw that you got your degree in Sally Struther's School of Medical Correspondence.
Personally for me, I'm lucky. I pay dingleberries for tuition at a state school, have really smart friends who motivate me to study and help me set standards, but are chill enough to be normal. They motivate me to work out/play sports, go out on the town, go out and date, BE NORMAL, while pursuing my own goals. It's sorta the perfect environment for me actually. In undergrad, it was a bit different. I stressed about looking good for the next level (getting into med school) and I sacrificed a lot (like my wellbeing) just to keep up with everyone/outdo them. When I got out in the working world after undergrad, I realized that NO ONE CARED where I went to school, other than my employer who hired me. But after being hired, it was up to me to perform to KEEP MY JOB, and not my alma mater. When I interviewed for med school, it was easy because people gave me the benefit of the doubt... but once in med school, it was up to me to perform on my exams/clerkships. Reputation can take you places, but it's up to you to cross the finish line (first).