In some ways, these things are indeed privileges of a sort. But not in every situation. For example, the height, thing. We know that people above a certain height are treated better, earn more, etc, etc. Yes it's inherent, we can't do much about it for that individual. But, just like other privileges, in this example maybe we can look at why shorter people are not treated as well, and, if it's some sort of systemic issue not based on merit, come up with solutions. It's not about knocking people down, it's about finding out why some people are not lifted up in the same way.
I believe it's more of an intersection of the two. We all know how much certain factors influence our patients (low SES, trauma exposure, learned helplessness, low educational attainment). It's actually integral to our practices on a daily basis.