Something else you should consider is the amount of small-group time and how big those small groups are. (I don't really know what scenarios would get you one-on-one time with profs... mentoring? Research? I guess.) At my school, even though we have 134 students (+ we have a bunch of classes with pharm students), our small groups range from 6-12 people, and we have these pretty much at least once a week. (They're usually 6 when we discuss patient interviewing or medical ethics, and up to 12 when we're going over respiratory physiology calculations.)
Aside from this, I totally agree that 100 vs 180 doesn't really matter. What matters more is how happy the current students are. I have a friend at a school with less than 100 students, but she feels like it is cliquey and not cohesive at all. However, I feel like my class is pretty tight and everyone is friendly towards each other, regardless of our typical "clique".