Schools won't look down on transferring to a more prestigious university. As long as your GPA doesn't drop, it's something that will definitely help you. Where you go for undergrad matters (especially if you're not a nontrad), but how much it matters is going to be different at every school depending on who you ask.
You should also consider finances- more prestigious schools will cost much more than public. You'll have to decide for yourself if it's worth it. My personal advice is that if you'll have to take out heavy loans, it's probably not worth it. You should also consider that you more than likely won't be able to maintain a 4.0 if you transfer. That's not to say you'll do poorly, but you can expect a slight drop. I knew quite a few transfers like yourself during undergrad and they all said there was a big difference in course difficulty after transferring.
More than anything else, you should consider happiness, as that's ultimately what matters most in undergrad. If you're unhappy where you are now, then I'd consider transferring, but if you feel like you've found a home, then I'd suggest staying.
Hope this helps-
Kevin W, MCAT Tutor
Med School Tutors