I did a primary care IM residency. Every year there were people who applied to all of the competitive subspecialties and matched (including myself). We were typically viewed as “applicant from uni of xx” not “primary care applicant from unv of xx.”
That said, how supportive your primary care residency will be of your fellowship application is another story. My PD was awesome and supported the residents in whatever they chose to do. I have heard this is not always the case…