I think there is a significant amount of variation between the practices and lifestyles of different types of sports medicine docs. From my understanding it is rare for a doc to practice solely sports med; rather, it is usually a component of their main specialty. So you really need to also like the main specialty as well as the sports component. I think each brings something else to the table:
EM+Sports Med: Focus on acute care of sports injuries
PM&R+Sports Med: Focus on long term care/restoration of function
Family/IM + Sports Med: injury prevention, exercise and diet prescription, treatment of concomitant medical illnesses (asthma, cardiac anomalies, etc), long term care, non-surgical tx
Ortho+ Sports Med: surgical intervention