This may sound bad but I think it depends in part on if you are male or female (mostly because of wanting more children). I am a married, mother of three. I have had the same thoughts as you ( except no more kids for me). My mother in law went to med school with one kid and had another on during fourth year, and another her first year in practice. I personally believe that you can totally do it and your family doesn't have to suffer (a lot anyways). My husband said he really doesn't remember his mom being gone all th time, or not being able to do things, etc. and he was 5 when she started med school. However, from talking to my MIL I think there are two things that will kepp you successful:
1) You will have to give up being ranked #1 w/straight A's. C's are OK too.
2) it is all about balance
Someone once told me when I started this process that if you are going to do this as a parent you have to live by this motto "One minute I am a student, the next a parent. Which it will be depends on the situation where I am needed most, and I can change lightning fast between the two." This means expect to have days where you HAVE to study, no excuses. However, expect to have days where sick kids, soccer games, etc. are more important and you HAVE to be a parent first. Next, regardless of if you are mom or dad, having a supportive spouse is absolutely necessary. I am lucky because my husband probably understands the life of med school better than I do. However, you spouse has to understand that this is a huge sacrifice for him/her too. But he/she has to understand that and not be resentful. My hubby and I laugh that "WE" are going to med school because it really is a family affair.
I think that this is a very personal decision. I KNOW that I would have regretted it if I had not tried. My grandma had huge problems with me going to med school in the beginning. Like I told her... " ya know, I may not be able to handle it, I may fail. Maybe it will be to much to handle and I will have to quit. However, I'll be damned if I don't at least try". This is only four years of your life, and yours kids. With good planning and balance it can be done rather unpainfully. Lastly, think long and hard about the specialty you think you may want to pursue. For me, I wil NEVER to surgery. Ther residency and lifestyle would not work for my family. I am looking at going into fmaily medicine or rehab. Both have humane residencies and the practice can be tailored to fit ith family life well. In the future you will just have to investigate the residency programs and later practice options really well to make sure you will be happy, as will your family. I also had the OMG what am I thinking after I was accepted. I actually had about a month of freaking ot and debating, etc. But I didn't make or attempt to make any decisions and just let it sit for about a month. Then slowly I got excited gain and got the I can do this attitude again. Now I am really excited.

GOOD LUCK with this decision!!!!!

edited to add: My last two kids are 18 months apart because we wanted three kids and I refused to have kids during med school or residency. That is just me, I can't imagine adding that to our plate. However, it can be done. As I said my MIL had one during fourth year. I think having kids in med school is easier if you are dad. Although still hard, you aren't the exhausted, engorged, breastfeeding, hormonal mess that mom is. During my pregnancies I was in school and found it very hard to study because I couldn't focus.