I started residency at age 36, so I was ten years older than my fellow interns who were trads. For the most part, it didn't make much difference in terms of how people treated me or at interviews, other than getting some ribbing from the young'uns. (I was the oldest in my residency class at a high power academic program by seven or eight years.)
But I will tell you this: as you get older, you just don't bounce back from those nights on call or night shifts the way the younger folks do. In college, heck, even at your age, pulling an all-nighter was no big deal. At my age (40), well, being post-call feels a lot like being hung over without having had any fun the night before.
As a resident, you pretty much have to suck it up no matter what age you are. But it's still doable, even for someone in their 30s. And once you're an attending, you have some ability to control your schedule and prevent that constant feeling of jetlag based on what job you take. Some people work in the outpatient setting, in which case you can simply not schedule any night hours. But depending on specialty, you still might have to take night call.
I've compensated for it by becoming a nocturnist (work all nights) so that I never have to flip back and forth between nights and days. That solution works for me since I like sleeping in late and don't have kids. Most of my fellow middle aged colleagues who have families prefer to suck up the occasional night so that they can be on their family's schedule most of the time.