Rankings do matter a bit. It's more about how well you do, getting honors, doing well on the USMLE, etc.
For better or worse, on average, kids with M.D.s from Haaaarvard, JHU, Stanfurd, Dook, UCSF and Washu probably do better in matching competitively than some of the unranked schools. But then again, there may be some selection bias here, the kids that get into Harvard or JHU usually have some phenomenal academic cred that other "unranked schools" may be unable to recruit.
Most of the top schools have better connections to NIH, research grants, very well-known faculty (who can write great residency letters) and a track record of producing academic clinicians.
Granted, if you want to be a PCP, it ain't going to matter all that much where you go or where you do residency. But if you're aiming to be the head of the pediatrics neurosurg dept at <insert famous hospital here>, go with the best fit and the best research school.
If you're lucky enough to get into Yale and Duke (or other similarly ranked schools), I wouldn't stress it too much, since the difference in rank between 5 and 10 is negligible.