You'd have to be somewhat crazy, even if you were only 22 years old.
There is no question that there are some hardships ahead if you pursue a medical career, regardless of your age.
A question that I was asked when I was contemplating a Family Practice residency over a Surgery residency because it was only 3 years was, "How old will you be when you plan to retire?" My response was about 65. The response was then, "Oh, so you'd like to work for 27 years at your second choice field of specialization to save 2 years of residency." Finishing medical school and residency at age 50 will still allow you to practice for 15 years or more. If it's what you'd rather be doing, it's worth going for. I'm definately glad that I'm practicing Surgery and not Family Practice (though I sometimes wish that I'd gone on and done Plastics after residency.) I'm in my mid-50's now, and have even considered doing a fellowship, but I am enjoying what I'm doing. (General Surgery is the redheaded bastard stepchild of the surgical specialties, though.)
One advantage that you have, though, that I didn't when I started as a non-traditional medical student at age 29 can be stated in 3 words. Work Hour Limits. The 80 hour work week is long, but it's much tougher to work the 100 hour work week.
Best wishes in the pursuit of your career.