It's kind of a tough question to answer, because many people who train at private practice type residencies don't apply for fellowships (or at least the really competitive ones), whereas probably the majority of those at academic centers do.
I think, in truth, it is up to you. It might be hard to get a fellowship at a really high powered referral center if you are coming from a smaller program, but it's certainly not impossible. If you impress your attendings, and show yourself to be a motivated and high quality resident, you shouldn't have trouble getting fellowships anywhere. As far as being close, well, that might be important and it might not be. It depends, if the folks at the center you are at trained at the neighboring large place, might be useful. If not, maybe not.
It also depends somewhat on your career goals. If you want to be an academic physician and work at a large referral center, it behooves you to train at one of these places. If your goal in getting a fellowship is to simply make you more competitive for a private practice position, it's not as important.
Like I said, though, a lot of it is up to you. Getting research grants can be done I would imagine, but it's a lot easier to do so if you are training in a place with lots of experience in this area. And getting fellowships is certainly possible. I think what your PD said is probably right though, it's more difficult. How much more difficult, I don't know. A lot of it is up to you and where your priorities lie. If it is on the location and the lifestyle, well, I guess your choice is obvious.