Of course, I can't quantify what "some people say".
It is pretty grueling, and I'm not sure I would do it again. But it's much easier than med school and residency, at least for me. On the other hand, you have more economic stress, because while you are now earning money, instead of borrowing it, you are now faced with paying off your loans and supporting yourself for real.
If you have your own practice, you will have new stresses: earning enough to pay the bills, pay your staff, the rent, insurance, etc. Getting insurance reimbursement. If you work for a large group, hospital, university, you have to meet all sorts of metrics to make the administrators happy ( see enough patients, utilization, quality, both real and perceived, etc )
Here's the real problem, as I see it, and the reason I think many doctors are disappointed with their careers, although I'm speaking just for myself here:
On some level, I probably thought that once I took the MCAT, did well in Organic, got into med school, passed Step 1-3, got into the residency, got the fellowship, passed my boards, and got the job, that then life would be easy, and I would get my reward. But there is no pot of gold at the end of the rainbow, no prize, no reward. When you're all done with all the training, you will just have a job. Or maybe a small business. It's a great job ( for some people ) it's a rewarding job ( for some people ) and it generally pays quite well. It might be the best job/career/profession there is. But it's still a job, with lots of obligations and stress. Again, keep in mind that many ( most? all? ) jobs and professions have similar issues. Life is hard, and making a living is hard, regardless of what you do.
You can mitigate some of the stress by choosing a specialty that you will be happy with and comfortable with, but it's still going to be a job, and there will probably be aspects of that job that you will dislike. All you can do to mitigate that is to choose your specialty and practice environment carefully.