Have you considered adolescent medicine?
It's primary care, there are no babies to kiss or baby talk to perform, the interactions easily split the gap between adults and kids, and the problems are rarely so dreary.
Obviously there's a fellowship, but I think there are plenty of opportunities within larger group practices to "specialize" even without the fellowship training. There are plenty of general pediatricians who do prefer baby talk and wouldn't mind being able to hand off their teenagers. Salary considerations are pretty much going to be similar to other general pediatricians.
If you are more academically inclined, a fellowship is probably required. But it may also lead to some other different opportunities depending on what your focus might be (ie STD's or sports med). You may end up working closely with school districts and their athletic trainers or the Health Department on various initiatives.