It's a meat market. It's insane. There are a huge number of people who attend, multiple sessions, multiple venues, and you have to camp out early for the popular talks/speakers in order to be able to get into the sessions. If you turn up on time, chances are you won't be able to get in. There are also some good courses that occur that you have to pay extra for that might be good to attend (as much cheaper as a resident than later). In addition multiple other associations have their conference or some kind of meeting during the APA including AACP (community psychiatry), AAPP (philosophy and psychiatry), SOBP (biological psychiatry), ISEN (neurostim/ECT), AAPDP (dynamic psychiatry), AGLP (LGBTQ psychiatry) and so on. Often programs have drinks receptions during it, there are lots of careers stalls, lots of social events, and the APA's caucuses all meet during it. The APA components also meet during this and they are open to APA members if you contact the chair in advance. Basically, there is a lot going on.
APA is my least favorite conference because it is sensory overload and there is too much happening. I go every year because it is a good chance for me to catch up with friends and colleagues, and I usually have commitments in terms of attending various committees that occur during the meeting and presenting stuff etc.
As a resident, I would definitely recommend going to it once (as its much much cheaper to attend as a resident or your program might pay), but the subspecialty conferences are usually better and more intimate.