I did pgy1 at a location with 1 other resident (there were 2 other programs in town, so 8 total residents and we did journal clubs and whatnot), but pgy2 at a place with nearly 20 residents total.
As a social person, I appreciated having the built in social circle. We aren't all close, but I have people to visit all around the country now. My network is that much broader. However, there was a lot of competition, especially for rotations. Everyone thinks they want to do critical or ID and those rotations are at a premium early in the year. I had very little mentorship due to the size of the program.
If you are someone who needs more attention, or likes to really stand out, a smaller program may be better for you. I worked with some very bright people, but I honestly couldn't tell you some of their strengths because it was just a blur of pgy1s.