Any good LOR will do several things:
1) Attest to your intelligence, intellectual curiosity, academic strengths, etc
2) Attest to your ability to work as a member of a team (What kind of a person you are to work with, friendly, helpful, rude, always late)
3) Attest to your motivation and interest in the field you are applying to (Pathologists, obviously, know what the field is like and are best equipped to evaluate this, but others may be able to contribute something useful. A surgeon, for example, may talk about your interest in pathology and how you were a great help in learning about these issues in regards to your patients).
4) Talk about what kind of a resident you would be (likelihood of completing assigned tasks, ability to work under pressure, working long hours, working under supervision, accepting and encorporating criticism)
5) Talk about what you are like as a person (People like to work with people they like. No one likes a brown nose, no one likes a snob)
6) Have some insight into your future career plans and how your qualifications stack up with these ideas (Do you like research, plan on academics, are you a budding teacher)
So, if you have a close relationship with a primary care physician, this might be a good letter to have for a pathology interview. However, pathologists want to hear from other pathologists who know what kinds of people become excellent pathologists and what it takes to succeed in residency. If you did significant research with someone, that would be an excellent suggestion for one of the three letters, or perhaps a 4th letter at some places.
I had all 3 letters from pathologists. 2 were from my home school and were people I met and worked with a lot during my PSF. The 3rd was from an away rotation I did early in my 4th year. I thought about getting a letter from 1) a surgeon I worked a ton with and who knew me from pathology as well, and who I talked with a lot about path-related issues, and 2) A primary care internist who I spent a month with and who could attest to my interpersonal skills, academic curiosity, etc. But I went with the pathologist, because while the other 2 could say nice things, the pathologist letters best related to residency issues. My dean's letter had lots of comments about my clinical rotations and interpersonal issues.