It matters somewhat, exactly how much is hard to say.
And its not really prestige. That can factor in, but there is more to it than that.
For example, prior history with students from the school matters. When I interviewed for residency, many residency directors commented on how impressed they were with prior residents who they took from my school. They can use that as a decision factor as it tells them something about the level of clinical training you received and what they can expect from you.
Another factor is having well known faculty - medicine is a pretty small world, and it gets smaller when you pick a specialty and apply to residency. If the school is well established, so that the faculty in a given specialty are well-known (either in the area or nationally) then your letters have more meaning to the programs you're applying to, since a letter from someone they know well will be more valuable than a letter from someone they've never met.