I have the very similar interest as you have. I want to eventually work in primary care getting to know and follow patients for many years. EM, Radiology, Surgery, Orthopedics, Anesthesiology, Pathology, Critical Care all are unappealing specialties to me because you will not have long-term patient care and follow-up. You only see patients for a short amount of time. That is another reason why being a hospitalist is a turn off for me as well because you only get to treat and see patients while they are inpatient and then you never know what happens to them when they are discharged. Now you have to take what I say with a grain a salt. That is how I personally feel about what kind of career in medicine I would like, but someone else can feel completely different.
One thing about pain management, and take it from a person who has chronic back problems who has seen several pain management doctors over the years. You will not see patients long-term. You are lucky if you see a patients 3-5 times before the treatment either works and they don't have to come back or it doesn't work and you have to send them off to either a neurologist, orthopedic, neurosurgeon, orthopedic surgeon, PM&R etc.
If you are like me who wants a long-term longitudinal relationship with patients that have chronic diseases your best bet is family medicine, internist, neurology, infectious disease, cardiology, allergy/immunology, dermatology, GI, oncology, nephrology, urology, hematology, pulmonology, pediatrics, rhemuotology, hepatology, psychiatry, OB/GYN and ENT for long-term getting to know your patient specialties and follow them.
For instance, I was with the same pediatrician for 18 years, my internist I've been with for 7 years, my allergist 21 years, and urologist for 10 years. Actually, I've been with my neurosurgeon for almost 2 years now so I guess complicated surgical specialties where you have continued problems, your surgeon can get to know you.
But if others love to work with their hands and cut people open, do more clinical research, etc then this lists will mean nothing to you and you'll be more interested in something like surgery or pathology, etc.