Yes, it matters for IM. If you want to do certain specialties like cards, your changes of getting that (especially for foreign grads, which I am not) are abysmal if you go to certain community programs. However, if you want to go be a hospitalist, then perhaps being at a residency in a certain area (such as southern CA, if you are looking for jobs there) is a big advantage (versus being at perhaps a more well known university program on the other coast, etc.). Part of it depends on what you want for your life, and what residency you are able to get anyway. If you can't get a big university residency program for IM, but you want to do GI or cardiology, you might have to then do a research fellowship for a couple years after, and do some networking, etc. Some of the smaller community hospitals don't have much ICU experience, which could hurt you as a pulmonary/critical care of cardiology fellow, if you want to pursue those areas...
But the others who posted on here are right - you may still get a lot of job offers after you are done, if you are willing to work in high demand areas, if you like other people and have a good bedside manner, and if you are a good workplace politician (getting on well with people who supervise you in residency,and those who work under, around and above you as an attending).