PM&R does in fact have a subspecialty in sports medicine, with their own fellowships, and PM&R residents are also usually eligible for FM sports fellowships (I believe FM is usually eligible for PM&R sports fellowships as well)
There are really two sides of PM&R--the physical medicine side (think musculoskeletal, more outpatient focused) and rehab (spinal cord injury, traumatic brain injury, neurorehab, etc.)
While I'm interested more in the inpatient rehab side of PM&R, I've worked with a lot of sports physicians (FM & PM&R) and here's the general advice I got as far as which residency to pursue if you want to be a sports medicine physician.
Ortho: Only pursue this path if you want to operate. Otherwise generally not worth it (long and difficult residency, salary is far better if you operate).
FM: If you want to be the team physician, and treat the non-sports stuff as well (ie, treat athletes, but not always athletic injuries). Most FM sports docs tell me at least 50% of their practice is still general FM--it's difficult to practice only sports for most of them. Clearly it's the most versatile path--FM docs are needed anywhere and everywhere, and probably one of the few specialties whose salary will be going up in the years to come.
PM&R: If you really only want to practice sports and MSK. Meaning, you want to treat sports injuries. You won't be treating a football player's depression or gymnast's amenorrhea. I think PM&R trained sports docs will know much more about sports injuries (half of your residency will be muscoloskeletal-focused), but you'll be less versatile than an FM-trained sports physician. And just as how FM sports docs usually still practice general FM, you'll probably have to still practice general PM&R as a PM&R-trained sports doc. But general outpatient PM&R has a lot in common with sports medicine.
Note that PM&R is a 4-year residency (not including the one year sports med fellowship), so there is one more year of training involved compared with FM.
If you definitely want to practice sports, I'd recommend pursing the residency that interests you more. If you couldn't practice sports, would you rather practice FM or PM&R?