It's to make sure you click with the program and it clicks with you. Even more important than residency fellowship interview are more about recruiting/hiring a colleague than a trainee because a large number of fellows will commonly end up joining the faculty at that program. So it's important not to eliminate any social pathology immediately to make everyone's life more pleasant.
It's also to give you a chance to ask questions that are never included in the glossy booklet, get the feel for the place and look around and decide if this is a city you could stand living in.
