I also dislike hospital medicine and particularly critical care medicine--but I still really enjoy learning about internal medicine and I had a lot of fun in residency. I went to a university program with a heavy primary care emphasis (although not a specific primary care track) and also a lot of emphasis on education and lots of exposure to subspecialties. A lot of people in my program's leadership are quiet, nerdy PCPs and ID docs and rheumatologists and that definitely influenced the program in a way that worked for me. I got a great education and I mostly work 60-70 hours a week on inpatient rotations and 40-50 hours a week on consults and elective. I traded away most of my ICU rotations to the cowboys in my program like jdh up there. You don't have to torture yourself in training to be a great internist (btw, internist means any internal medicine physician, but often refers to IM PCPs). Ask a lot of questions about the culture of programs when you interview, because I think you would like a program like mine but maybe not one like jdh's.
P.S. Regarding family medicine, if you're okay with only practicing on adults, I think you'll be a much better PCP with a background in IM than a background in family medicine. But that might be less true if you're looking at those old-school IM programs where you spend all your life stuck in the hospital and clinic is an afterthought.