I would say it is not COMPLETELY arbitrary. What you say above is probably true for for those going into private practice who want to stay at a particular location, but for academics (and PP on a national level) it does matter. Programs with good reputations have broad alumni networks across the country. Look at the institutions you are looking into- where did the faculty train? A program like BWH has a large number of alumni as faculty across the country (same for Penn, MGH, WashU, UCSF, etc etc etc...), meaning when you want a job there they will put you above others, all else being the same. Similarly, others who did not train there know of the reputation of those places and know you are likely well trained. From what I've seen, fellowship interviews don't do slide reviews (or job interviews at academia), meaning they have no real way of assessing your skills and will base their decision on your charm and recommendations (and research talk for academic jobs). If recommendations are coming from famous people or your interviewer's old buddies from institution X that will have more meaning than random professor Y.
That being said, there is history on this board of talking about the "top 5" programs, and anything outside of that is crap. This is BS. There are programs out there people will say have great reputations, without much debate, those with great reputations and some debate, and those most would not consider the top. I personally would think the number of "elite" programs is like 10-20. All those programs the OP listed are probably in that category, and likely Pitt and CC too.