I think there are a lot of reasons. Commonly, people talk about the "eternal rounding" and of course there is some truth to this. I honestly have rounded for 5 hours before, but the reality is that on average, for a full census, we rounded for typically 2 to 2.5 hours. Some attendings were much better than others. This is not at all different from surgery at my school, where they may round for 1 hour in the morning, but also tended to round for another hour+ in the evening. The main difference was that the attendings never rounded with us on surgery, which was a major light at the end of the tunnel for many people who just couldn't bear it.
Another thing people talk about is the very esoteric, sometimes archaic, discussions on IM rounds. I've found that if you know your stuff, you can always control the pace of the conversation (that is if you are being spoken to). This is especially true with electrolyte abnormalities like hyponatremia.
Some don't like how in many fields of medicine you don't physically intervene to correct a patient's problems - you often rely on pharmaceuticals, fluids, etc...
Others get bored with the paperwork.
The big difference I've noticed with attendings in surgery vs IM is that IM attendings really don't have much going on after rounds. I like this prospect a lot. I've always wondered what they do all afternoon (clearly read articles, work on research/grants, see patients in clinic, etc...). I'm sure some of them are out there mountain biking, surfing, etc...though.