Yup. You need a strong, strong reason and a similar tier acceptance already. Med schools, especially top med schools, are a bit clubby/cliquey. Yale is going to put their nose up at any student who isn't at a top tier med school. Based on my interview there, the administration is already salty about most of their residencies being significantly less competitive than the top ranked programs in Boston and NYC. It's extremely rare for this to happen in med school. I think it's more common in residency, but even then, you have to be in the "in" club to make it happen.
I do think there's a strong element of favoritism when it comes to top tier school applicants, tbf, and it's not just from better opportunities to craft a resume. We all know that med school admissions can be a roll of the dice. Two of my good college friends were effectively the same applicant on paper. One of those roommates had a more compelling story and got into a T5. The other went to a more average med school (think ~70ish on USNWR). I kept in good touch with both of them. The T70 med student actually had similar grades (on a much harsher grading scale), more/better pubs, and a higher step score, and they both went into medicine. The T5 student wound up interviewing at 3 of the Big 4, where they matched, and had a laundry list of T20 choices. The T70 student didn't get a single Big 4 interview and only a handful of T20. They matched very respectably, but the floor and ceiling was so different for them.
It's just hard not to think about that when I think about top vs. mid-tier schools. I still remember getting brunch with both of them right around that time and remembering how empowered vs. discouraged they were regarding their choices and chances.