I agree.
First priority is choosing a residency that you'll excel and be happy in. No matter where you train, you'll be just as "prepared" for a toxicology fellowship.
But, if you wanted to do research or side projects in toxicology, you would give extra consideration to programs with affiliated fellowships. If you decide to go a step further, I would look for fellowships with more faculty/fellows. More faculty and fellows most likely will mean more possibilities you would find projects and a mentor that fit. But, that's about as much thought as I'd put into it. It's hard enough deciding where you want to train, than worrying about fellowship training afterwards.