Yeah, kids understand perfectly why their parents work so much. 🙄
There are a lot of things kids don't understand. They don't understand divorce, but sometimes, divorce is in the best interest of the parents AND the children, for their safety, happiness, well-being, etc. Same with career choices. If you really feel you won't be happy or fulfilled doing anything but medicine, you owe it to yourself AND your children to find a way to do it.
Making decisions like this based on whether your 5 year old will not understand why you work so much is your choice, but there are many people out there who make it work, and their children survive and are not scarred for life.
Can you do surgery or critical care and make it work? Probably less likely than if you chose primary care. Rads, optho, derm--great choices for lifestyle, but consider that you pay your dues for getting into those specialties up front. I know of someone who actually sent her toddler to live with her grandmother for a year so she could focus on her studies and land a derm residency (and she was married). Hmm, maybe not in the best interest of the child??
Intern year sucks for everyone. After that, consider that you only have 2 years left if you choose general IM, FM, or peds, and your load lightens considerably in those last 2 years at many programs. Furthermore, you have a much more flexible schedule when you do start working.
It's all about time and money. You can have one or the other, rarely both, unless you win the lotto. People get all freaked out about paying back their loans. This shouldn't be a problem for ANYONE making more than $120K a year. Yeah, you might not get the beemer your first year out, but if you are going into medicine for the fabulous money, you better be prepared to have less time with your family. If you value time with your family and don't mind a lower (but still very comfortable) salary, choose something in primary care.