It is certainly tougher with a family... I think since our specialty requires so much reading, you can definitely plan ahead, as compared to being in an inpatient, non-clinic specialty, where you never know what to expect.
If you read in evenings, during breaks in the day (have stuff printed out, or UpToDate topics, or parts of the texts), on the can, etc., you should be able to get by with minimal weekend time. If you like to do fun things in the evenings on weeknights and not read as much, you may need to spend many hours on weekends catching up.
I don't have a family, but I am not breaking my back, enjoy my social life, and am progressing nicely. During the first few years, I did read a fair amount on weekends and evenings, but not enough to make me miserable. Now, it's mostly during the day in between patients and a little in the evening. Even us barely-matchers do okay, just make sure you make the most of the time you do have.
-S