At the very minimum, wear a sports coat. Borrow one from a friend, family member, etc. Or pick one up from a thrift store; you'll be surprised at what you'd find there.
Without a coat, you'll stand out like a sore thumb, at least at the schools where I have interviewed.