I've definitely heard some downtalk if you're an ivy leager or other high tier school attending an interview at a much lower tier- the reactions automatically jump to the conclusion that this particular school may be just a safety school. However, I have yet to see any of this myself, and havn't heard too many stories of an interviewer specifically pointing out a portfolio's insignia (mine is stamped onto a leather cover, without color so its unnoticeable- perhaps yours is like this as well?). I say bring it, there's no shame in having a portfolio with your school's insignia on it- unless of course, you were from Ohio state and applying to Michigan schools and vice versa
(goes with any other rival sports states/schools)