All good advice from above posters.
I'm FM FIRST than Sports Med. Theres a reason its called PRIMARY care sports medicine.
Most academic places when you apply for a job want you to do that for that for their athletes. They have orthopedists for ortho needs. They dont want you to looking at peoples knees. This is where your primary care part comes in.
For example, my program, D1. My attending (PD), and the head of ortho are the guys that travel with the teams when they play. My PD (although super good at ortho), isn't the one being called by the athletic trainer for a suspected ACL.
In training room, you'll be called for the "medical" aspect of the athlete more than the orthopedic part. So if you're "base" training didn't give you any of that.. I feel you can't do much. As i'm in fellowship now, this is how i'm finding out committee goes through things.
So, statistically, and training wise, if you feel FM is something you will like doing (even fulltime, if things don't go the way you plan), than that's better to eventually give you access to PRIMARY care sports medicine.