Actually, fellowships are NOT privy to in-service exam scores unless you tell them yourself.
Who you know does matter, and the truth of the matter is that Duke and Baylor et al do not have a monopoly on important contributors to particular fields. Certainly, if you're at a major center, you'll have a better chance to meet "important" people in a wider breadth of fields, but if you have a particular specialty that you know you're interested, you may not meet the absolute biggest names (or be in a situation to build a significant relationship with them) if you're at a "big name" program.
For example, one of the lead authors in the study that proved the effectiveness of DMARDs in RA is the residency director at a good, but not "name", internal medicine residency in the Midwest. If adult rheum is your destiny...might be a good guy to know because every rheum fellowship is going to know him. If you're interested in Pediatric Critical Care or Peds ER...the lead author for the 2010 PALS guidelines is an attending at a very good, but not "name", peds residency in the South - another one of those people who, if they call on your behalf, gets people's attention.
I agree with the recommendation that if you feel that a specialty is in your future, if you're an M4 interviewing for residency, it's wise to take note of how many people they put into fellowships every year. For medicine, there are ~ 3000 fellowship spots in the Med Specialties Match...while there are about 5200 categorical IM spots. So if you see a program that doesn't consistently put at least 50% of their graduates into fellowship, it should make you wary of the program and their focus - it can still be a fantastic program, but it might not be what you're looking for. In peds, there are about 2500 categorical residency spots each year and roughly a 1000 fellowship spots, so again, being in that 30-40% range is a good sign that they'll be a good launching point.
As someone who is going into fellowship and at a residency program that has a fair number of fellowships that it sponsors, I think it's a good thing. One you're more likely to have faculty engaging in research projects who are used to mentoring trainees, you have fellowship directors - even if not in the field you're interested in - who can give you insight to what fellowship directors want, and you have the fellows themselves who can serve as role models and resources having been through the process. Obviously, you want to make sure your residency education is not compromised, so going to a place that has fellows but is not fellow-driven is important, but I think it's a very valuable component to have fellows around.