It appears that you've been interested enough in dermatology to invest a large portion of time, energy, research, etc.. Now... maybe you've been immersed in dermatology long enough to start to see some of the negatives...giving you mildly disillusioned look at your future specialty. That's something most medical students never have a chance to see and only realize later on once they're in residency. We all have those "the grass is greener on the other side" types of thoughts, but EM is no different in that we have our own set of negatives along with the positives, you just haven't had a chance to see them yet. Just do a search and you'll find plenty of us talking about those. I'm not done with residency yet, but I think that most of us that do this, really enjoy the specialty, recognize the negatives, but can't really imagine doing much of anything else. I might come in bitching about a bad night, but when put to the test... I really wouldn't pick anything else. Honestly, I was never even tempted to do much of anything else.
I'm with EM_Rebuilder on this one in that walking into a 9-5 office with shirt and tie, overly groomed, doing dermatology, week after week, year after year, is the stuff of my nightmares. Sure, I like EM and all the variety, procedures, occasional adrenaline moments, etc.. but I also find the drunks and drug seekers humorous, the crazies interesting, don't mind being yelled at by patients, don't mind occasionally having to yell at consultants, etc.. I hate sitting still or moving slow and am incredibly bored when I am.
You strike me as someone who is fairly idealistic in their approach to finding that "special specialty" that they feel is perfect for them, but you won't find it. Each has their own drawbacks and let's face it... after 10 years, it's going to me more of just a job than anything else and you'll be focusing more on the things you do outside of your job.
Derm schedule is going to be easier to control, period. You're looking at raw hours worked in EM, but your time off is not true time off. You're going to be working some nights, weekends, weird shift hours that are not very compatible with your wife, friends or families' schedule and. Yes, I'm sure you can make it work, but if lifestyle is important to you, I think few would argue that Derm gives you a far better lifestyle.
My advice? Just do Derm. Your wife and family will thank you for it. I have a feeling you'd be one of those that switched specialties 1 or 2 years into it and that always sucks for the program.