Honestly, I think the main reason you don't see a lot of people switching out of psych is that I don't think any of the other specialties will give you credit for the PGY-1 in psych, so you will probably have to redo your intern year if you switch out of psych. In contrast, you can get credit from psych programs for having done a PGY-1 in medicine (making it easy to switch from IM or Neuro into psych). Nobody wants to redo their intern year, so that is a strong incentive to stick with psych after you pick it.
Still, switches do happen out of psych. Someone who used to frequently post here a few years ago wound up switching from psych into IM or FM (I don't think they ever posted about why though).
Psych is perceived as an easy specialty by many people, but there are definitely aspects of it that can be draining or very unpleasant. Non-compliance with treatment, manipulative behavior, and poor hygiene are things that many doctors don't enjoy dealing with from patients and psychiatrists see more of all those things as part of the nature of mental illness. As a med student, you may not be as irritated by the homeless people who feign mental illness in an effort to get a place to stay as you might be once you are the resident who keeps getting called about them.
There are also apparently psych programs that make their interns work just as hard as IM residents. As part of the decision making process, I think it does help to ask yourself if you would still want to do psych over IM (or some other field) if you had to put as many hours into psych as those other fields.