I honestly can't imagine that being the issue that makes or breaks you. I can imagine that there would be some benefits to it--like additional breadth to your experiences as an undergrad. Honestly, I think you would be just as likely to come across a prick that cared as you would be to come across a person who attended different schools in the summer themselves, and found it rewarding. If you want to do it, I say do it.😉