I interviewed at both schools, and was originally really attracted to Downstate because of the clinical exposure and patient experience, just like you said.
But actually, when I interviewed at Downstate, my interviewer spent a good amount of time explaining to me that the diverse patient population part isn't really true. He was like, "yes, we're across from Kings County Hospital, which is a huge plus, but our patient population is not diverse at all." He basically said there is a huge Caribbean American population in the area, so you would see a LOT of sickle cell (which is what he studies) but, on the other hand, you would never see a lot of other things (Cystic Fibrosis was one of his examples).
This is just one guy saying this so maybe it should be taken with a grain of salt, but he was basically saying the place is not all its cracked up to be-- which, obviously, seemed like a bad thing to come out of an interviewer.
He did say that the school itself was super diverse though, and that it was great to be in that kind of community.