I did a combined IM/Psych residency and have worked in both specialties. Psychiatry, like IM, can get old fast if your patient base lacks diversity (socioeconomic, cultural, etc.). In other words, if you practice in an exclusive suburban area, you'll see many middle-aged women with MDD or depressive D/O NOS; teenagers with ADHD, ODD, conduct disorder, and parent-child relation problems; occasional D&AA/SA issues; etc. If you practice in the inner city, be prepared for a plethora of substance abuse patients and antisocial PD patients. If you work in an inpatient facility, you'll practice more on the medical side and see a larger population of people with significant pathology (e.g., chronic undiff schiz, parnoid schiz, neurleptic malignants, etc.).
Psych is a great speciality, but it CAN get old fast. This is true, however, of any specialty. I worked as an internist for a year and found myself treating the same types of patients over and over again: NIDDM, OM, bronchial asthma, obesity, COPD, DDD, DJD, fibromyalgia, OSA, etc. Meh, it got old fast.
Your question is actually very subjective. You are the only that can truly answer it. How others feel about you as a future psychiatrist is irrelevant. How you feel about yourself and your work is what truly matters. If some neurosurgeon gives you a flippant or condescending comment about your chosen speciality, what do you care? You have undergone the same amount of training as most other specialties (e.g., peds, IM, general surgery, etc.). You are still an MD. You save lives. You change lives. You impact lives. As a psychiatrist, you will make a difference.
Now, having said that, I will admit that many physicians, health care providers, and laymen will claim that a psychiatrist is nothing more than a pill pushing head shrinker. If all you do is Rx and do med reviews, then maybe there's truth to that statement. I do more than just 10-15 minute med reviews. I do some therapy. I round at a few hospitals. I do some part-time work at an ER. I also work doing IM at a local urgent care on a contingency basis (to keep my clinical skills honed). Variety keeps things interesting and keeps your skills sharp.
Medicine in general, not just psychiatry, is experiencing many problems. I've become dissatisfied with my job as a physician, which is why I attended law school and am transitioning into law. If you can, try to do a dual IM or FP/psych residency; give yourself options just in case you decide it's not for you. No matter what, don't let the opinions of others affect your decision. Psychiatry can be very rewarding, both professionally and financially.