The biggest factor is probably the liability, but there are other reasons as well. For a group of people (radiologists) who already prefer minimal patient contact, having to deal with scores and scores of women who either have or are afraid they have breast cancer can be too much. Other than that, the work can be extremely tedious, and the diagnoses and recommendations are very algorithmic. You need not necessarily have very good critical thinking skills to be a good mammographer.
There are some upsides though. There's plenty of patient contact, if that turns you on, and there's a nice mix of modalities. Depending on how modern your department is, mammography is typically the last bastion of screen-film radiography, which is sorta neat to see in practice.