1. Everything depends on your physical, not chronological age. I know 25 year old obese smokers who are too old for things like surgery. I know a few 50 year old jock/ mountain climber types who are going to be able to whip your butt in any athletic endeavor for the next decade or so, who wouldn't have any problem.
2. Some older people don't like to be bossed by 25 year olds. Surgery is much more regimented and military in structure, so you have to be able to see your senior as the boss even if s/he is half your age and only a year more experienced. This is true in all fields, but more so in surgery and for a Longer residency.
3. Sometimes if you are older the shorter residencies will be more appealing. You will hear a Lot of nontrads say, if I started at 25 I would probably have done X. You only have so many years to practice so most people like to get out there and be a doctor before they retire. Also you probably are able to practice well into your senior years in the fields where physicality and steady ands don't matter as much. You will see more 80 year old psychiatrists out there than orthopods.
4. At the end of it all, do what you enjoy. You usually don't know until 3rd year of med school when you see more things. But mny people are influenced by the mentors you meet long the way and the various personalities each specialty has. No specialty is off the table, but doubtful you really know what each specialty entails until you get there.