You will likely get a lot of different opinions on this, because literally EVERYONE has a different career set-up.
However, I would say that in terms of neurology this questions depends most significantly on your choice of sub-specialty (or lack of subspecialty). People who work standard clinic hours tend, on the whole, to have better schedules than those who are continually "on call" or in-house. However, even with that distinction you will find very busy private practice neuromuscular specialists or generalists who work just as hard -- or harder -- than some neuroIR guys/gals or ICU specialists.
I guess the short answer is that if you aren't super money-driven or super-academic, you will have the opportunity in neurology to have a good lifestyle when all is said and done. I haven't said much here, though, as this is essentially true in any specialty.