I know some FM-sports med docs who do only sports med and some who do a little primary care lite on the side (no complicated people with lots of comorbidities, etc.). I know of one who does sports med + outpatient primary care + inpatient care for her own patient panel. Regardless you'll obviously have to do FM for a few years during training.
Also worth bearing in mind in mind that most sports med patients, with the exception of some of your old folks with arthritis and broken hips, are on the whole a fairly healthy population and won't need much in the way of chronic disease management.
PMR without sports med training, at least in my region, is a bit of a different animal than FM+sports med. Our sports med and ortho guys do most of the ACL tears, tennis elbows, concussions, fractures, arthritis, etc., and the PMR guys focus more on rehabbing folks with strokes, MS, Parkinson's, etc. to work on improving functionality and quality of life. YMMV based on where you end up obviously but at least around where I've trained so far it's a very different patient population.