I was torn between PM&R and psych as well. The same thing made me shy a bit away from psych (people not thinking I'd be a "real" doctor, and family opinions). I mostly got over that when it was really time to decide, but when it came down to it I had just a slight preference for PM&R because I liked the inpatient environment there much more, wheras in psych I definitely preferred outpt but felt I'd get a bit bored/lonely. I'm introverted, but I need some "coworker-type" contact. And I really enjoyed the interdisciplinary aspect of inpatient PM&R (same for inpatient psych--I just didn't enjoy working with inpatient psych pathology/patients).
I probably overthought things too much. But I felt I could enjoy about 4-5hrs of outpatient psych at a time, compared to 8-10hrs of inpatient PM&R. I did psych really meaningful (especially addiction medicine). Same for PM&R. Eventually I did a whole contemplative hike to try and figure out what to do and it hit me I smiled more on my PM&R rotations, so I applied PM&R.
Here's what I can say: go into the specialty you enjoy. But realize if you get it wrong, it's not the end of the world--you can jump ship (with some difficulty) and switch while you're a resident. It may set you back a year or two, but that's nothing in the grand scheme. We're talking about the rest of your life, and it's worth being happy.
That said, perhaps you could be quite happy in both specialties. If that's the case, one thing I didn't think at all about when I was debating the two was how easy it is to work literally anywhere you want as a psychiatrist. I could rent a room in an office and start my own practice--schedule my own patients, etc. It'd be so easy! You cant do that in most specialties. I could set hours like 8-2pm M-Th. I wouldn't be super rich, but I'd probably still do ok.
The location thing actually becomes more important when you have a family, which is when it hit me. When my son was born our priorities changed and I went from looking at big-city VA SCI jobs to wanting to live near grandparents. We got really lucky--there's an inpatient rehab unit 15 minutes from them. Otherwise the next one was a few hours away. The unit near them was so small it only needed 1-2 physicians, but lucky for me Coastal CA is hard to recruit to (expensive, not many jobs for spouses since it's not SF or LA), so they had an opening.
So it all worked out really great in the end, but if it didn't I'd probably regret not going into psych for that reason alone. I'm really happy in PM&R, but I still think I could've been a bit less than really happy in Psych. Happy is happy. I'm at the point in life where I would gladly accept a little less job satisfaction for the additional family ties/"life satisfaction" that came with being near family.
