As a 55 year old 3rd year medical student who has traversed many murky waters on my venture through medical school, I say that if someone truly wants and has that burning desire to be a doctor, they should definitely try to do so. As others have already said, you will be the same age in about 6 years, whether you go to medical school or don't. . . and if you really have the desire to become a physician, you won't let your age stand in your way.
As to the comments regarding your classmates ridiculing you, that may be true in some cases. However, out of the majority of my class, I can only think of 1 or 2 who may have felt I shouldn't be there. Almost all of the younger students were very supportive, and some even said they got a lot of encouragement when they were feeling overly tired by seeing that I was able to complete the task at hand.
As to medical issues affecting older students, of course that's a possibility. I know I've had my share. . . but they haven't stopped me yet! And anyone, regardless of age, can have a medical problem. Whether its a new diagnosis of a condition, or possibly an accident that has incapacitated them; there is no saying that an older student won't make it through just as easily as a younger student would due to such issues.
Long work hours. . . you bet they are. During school, you spend a great deal of time studying. During rotations, you may spend what seems to be an interminable amount of time on your feet working. But, somehow, when you need to do it, you just do. . . especially if it's something you seriously want to be doing. As a matter of fact, during my first internal medicine rotation, my attending was going to be gone for a few days and said I could take the time away from the hospital if I wanted. But, unlike my colleague, I told him I would be there every day to check on MY patients. This is a very important journey to me and I love every minute of it! My colleague, who is about 28 years younger than me, said I was simply 'anal' not to take the time off. I told her that if that meant I cared about my patients, she was right. And I won't ever change that!
Anyway, to the OP, you are NEVER too old unless you think you are. Go for your dream with all your gusto and let the naysayers fall by the wayside!