It's funny. I had a similar problem when I told some docs that I was applying to medical school. Then, when I said I was thinking of Pathology, many of them gave me similar reactions as you describe. However, I found that most of the negative comments came from docs who really don't enjoy their jobs anymore for one reason or another (none of them Pathologists, either).
On the other hand, the pathologists that I work for are all very supportive, and have given me nothing but encouragement about going to medical school, and about my interest in pathology. They tell me that Pathology needs good people, and are extremely willing to teach, or provide advice.
Don't let anyone get you down about your career path! The important thing is that you choose a specialty that you love and gives your life meaning. 🙂
Brian