I too felt the same. I had an easy solution. By the time I discovered how cool FP could be, it was too late--I hadn't applied into any FP residencies, only psyche!
One frustration I have with psyche although I do not regret my choice is that psyche doctors often forget their medicine--to the point where medstudents know it much better. It can be frustrating. E.g. order a med consult just because the patient coughed once.
If you want to treat the "whole patient", such as have a good relationship with the patient that will last years and involve treating their physical as well as psychiatric problems, and want to treat entire familes, FP may be better for you.
Psychiatry involves specialization in one field. In Psychiatry you will see a lot of negative things you don't see in FP, e.g. in urban settings you get a lot of malingerers, recurrent drug abusers who have no desire to quit their addiction but keep coming to the hospital for free food and housing. If this ticks you off and you can't handle it consider going to FP.
Now I didn't go to FP so I can't tell you the negatives in that field but I can imagine that because several fields are in FP, it can be difficult practicing in so many areas.