For many of us, deciding on a specialty is more about "ruling out" what you don't like, as much as "ruling in" what you do.
The first thing to decide is how much patient contact you want. If the answer is "not much" or "not awake", then forget radiology, pathology, and probaby anesthesiology. It sounds like you've already ruled out the surgical specialties.
It also sounds like you want fairly sane hours, or at least limited hours. Reasonable schedules are doable in most of the medicine and primary care fields given the right practice environment, though obviously easier in others.
PM&R is a good choice for those who are low key, like being relatively independent, enjoy taking care of relatively defined problems with a defined end-point, and also like to work at a somewhat slower pace with clearly defined hours. It pays pretty well (from what I hear) and the lifestyle is very good. If your personality fits, and you enjoy the work, then it's hard to beat.
EM shares several of those characteristics, but tends to attract more active people. EM has virtually no continuity of care, doesn't necessarily involve definitive diagnoses (but then again PM&R isn't really much of a diagnostic specialty either), and always involves some night shifts. Those are the down-sides, as I see them. The people in EM tend to be very smart attention-deficit kids or at least have some tendencies in that direction. You'll find lots of adrenaline junkies in EM--at my residency, the favorite pasttimes were mountain biking, rock climbing, and snow skiing. One guy was really into ice climbing. More EM physicians tend to ride motorcycles or drive fast cars than other specialties in general (except maybe trauma surgeons, with whom we share many common traits). One of the guys I work with is also an active duty cop. He owns LOTS of insanely fast vehicles (he offered to sell me a couple old Ferraris), which he can (of course) drive very fast without fear of getting ticketed.
A good ER doc has to be able to laugh at life's vagaries, sometimes even when it hits you in the face (or vomits on your shoes, or tries crawling to the bathroom too drunk to walk). You have to be able to roll with the punches, and find them really amusing. One poster noted that EPs tend to view medicine as a job, not a calling, which I find to be true.
Me, I work about 35-40 hours a week on average, and don't really feel a driving desire to work any more than that. If I get a raise, I may cut back my hours further still. More time on the beach for me!