OP, you've said that you would be more comfortable working with underserved, minority patients, but you have to realize that it is these patients that really need a "people person" doc. I've worked closely with underserved patients, many of whom are non-compliant, and the biggest reason for their non-compliance (besides inability to afford medication) is that they don't feel comfortable with their doctor. I would take that into consideration before saying stuff like you would much rather work with underserved people than affluent people. Have you worked with underserved people previously? Did you feel like you could communicate with them better? If so, why is that? Do you have something in common with them? You don't need to answer these on the forum, but think about them and maybe you'll be able to pinpoint why you're having difficulty in your current position working with more affluent folks.
Also, I've worked with both pathologists and radiologists. In fact, in the breast radiology center that I shadowed at, these two specialties have a weekly conference where they lecture each other and discuss findings, and they call each other constantly to talk about patients. Some subspecialties within radiology have a ton of patient interaction whereas regular, diagnostic radiology really doesn't (though this depends on the individual radiologist). Do you think your main concern is trying to talk to people whose knowledge base isn't the same as yours (i.e., patients), or are you concerned about talking with anyone? I know some people, myself included, have a hard time explaining things in simpler terms to people.
I also second everyone's suggestion to get a job or volunteer position that will allow you to build relationships with people, like tutoring or some other service where you're seeing many of the same people each day/week/whatever. You'll have practice with small talk, but it'll be a familiar set of people so you'll feel a little more comfortable interacting, hopefully.
