You will probably hear a huge variety of responses to this one, but I'll give you feedback since I came from a heavy primary care-weighted school. I think it DOES make a difference, and I'll tell you why. Our program makes outstanding primary care docs, because the curriculum is designed to do that. Our exposure to the specialties was slim, and none in some. And they made it difficult for you to get the exposure you wanted. HOWEVER, with that said, I will say that in the past 3-5 years, our grads have done well finding specialty spots, and this year we already matched urology and ENT for 100% of our applicants. So I think the opportunities are there, and your school will NOT hinder your ability to get a spot. But it might make it a bit more difficult. For example: if you come out of Yale Med, you likely have had exposure to what you want, and you've most likely been given some great research project in your lap for you, and you might have written a thesis. These are extremely difficult to get involved with at my school.
So to sum it up. I think you can still easily do what you want with hard work. I do think it's a bit easier at a school that is not so primary care-weighted. Stony Brook grads do go on to specialties. You just might have to shine a bit more as a student.