I think the downsides vary for each person. I think dealing with inadequate resources/systems for patients, stigma against mental illness, and difficult patients are things that are hardest for me. However, the up sides are numerous!
In addition, there is a large variation between the percentage of time that each person spends doing psychopharm vs. other things. Sure, there are some psychiatrists who prefer to do psychopharm (but even then, there is a range of what people might choose to specialize in (mood disorders, psychotic disorders, etc.) and what populations they decide to work with (child/adolescent, geriatrics, forensic, inpatient psychiatric units, consult-liaison, etc.)
Some psychiatrists go on to do mostly therapy (and many of these do extra training in analysis), and there are many that do therapy a few hours a week, or do combined medications and therapy. Others do purely basic science research, work in administration, or teach, or combine these things with therapy and clinical work.
I see a lot more of my attendings having the time and flexibility to maintain involvement with varied interests than in other fields, where docs may not have the time to do other things.
So in sum, my view of the field is that some psychiatrists enjoy spending most of their time managing medications, but the possibilities for fulfilling work in psychiatry are much more varied than it might seem at first glance.
You'll get a better sense once you do your rotation, and it would help to ask your attendings what else they do once you get there. Good luck sorting our your career decision!
