Why is "dinner" in quotes? Are you expecting that they are really planning something very non-dinner like?
I have been wearing "slacks" with a "button" down shirt. No "tie". Can't go wrong with that, looks "nice", not overly "formal". If "people" overdress you'll be in the "ballpark" of more conservative dress, if people underdress you'll still look comfortable.