As an immigrant I believe I have a unique perspective on this considering that my entire life I didn't hear a single word about "elite" or "second-tier" or any of that stuff until junior year of high school when it began to raise blood pressures among my peers and - admittedly - myself to an extent.
The perspective is this: it really doesn't matter.
I'm not going to lie to you and say the prestige of the school you attend has zero effect on your career (
@SouthernSurgeon has already explained that) but I will tell you that school prestige is totally unimportant for you as an individual.
How do I know? Because you didn't do it. At one point, consciously or not, you decided that the sacrifices inherent in achieving something of that ilk did not have enough value for you and I imagine you were not born into a position where it could be handed to you. You know that for the things that matter to you, achieving a top 5 acceptance was less important than other things. Getting into medical school in this country is an enormous achievement (much less MD, much less Einstein), something one ought to be proud of. The number of people for who this achievement is insufficient is vanishingly small and I can almost guarantee that they are mostly deeply unhappy people.
To "make a difference" means many things. One can "make a difference" coming from any background. There are DOs and FMGs on the faculty lists of Harvard and Stanford Med. There are physicians from no-name schools and Harvard alike somewhere in the boonies providing much needed care to those who would otherwise have no option. There are people from no name schools from countries you can't even point to on a map right now making a difference in a foreign territory providing care. There are MDs who had no idea what to do with their lives until 35 that are important parts of their respective teams.
Yes, one should care very much about one's education. Yes, one should care about one's training. However, I think you will find (if undergrad didn't make this readily apparent already) that education and training have very little to do institutions and more to do with you. Institutions provide opportunity, support, facilities, and frameworks but ultimately it is up to you to determine what you feel is important, what "making a difference" means to you and then capitalizing on it.