If you actually want a life outside of medicine, go with Derm. But you'll have to kick ass in med school, do research, get AOA, rock Step 1 of the USMLE (245+), do a bunch of other stuff, and you might match. The lifestyle is what makes it competitive. Fairly easy residency once you're in, 40 hrs/week for life, No call, low stress, great pay, Medical nirvana essentially.
Stay away from EM if you want a good family life. Burnout is very high. You get sued more than you'd care to think about, and it's pretty high stress. Very high risk, and not worth the pay IMHO.
Optho isn't nearly as popular as it used to be, but it's still competitive. Primary reason that it's a little less competitive now is probably decreasing reimbursements (moreso than most specialties), and cannabalization of certain procedures by optometrists. These cats used to make serious bank back in the 80's. I'm talking 500K, now it's maybe 40% of that for most, which is still a hell of a lot of money, but I guess most were just used to that kinda cash flow. Family life is pretty good.
FP can be family friendly. Can also have long hours. Really depends on your patient base. Pay is pretty low cause you're not doing any procedures, so Medicare, medicaid, and all the other insurers like to crap on you.