I don't think MS3/MS4 were worse than MS1/MS2 but I can tell you a few reasons why they suck.
You change rotations every 4-6 weeks, so as soon as you get used to something, you start doing something completely different, in a different specialty, often in a different hospital, with completely different people, etc. So basically you feel stupid the whole time, and need to adjust to a lot of changes very quickly and once you adapt you start all over.
You are working full time, which would be completely fine if you got to go home and enjoy life and sleep. However, once you get home you have to study for an exam that covers all of that specialty. That exam is a huge component of your grade and will determine whether you pass/fail or honors, which is a huge deal while applying to residency (for some specialties more than other but it definitely matters). You're so tired at the end of the day that it is super hard to study, so you end up spending your precious few days off of the month studying or you end up risking not honoring.
You are not only evaluated on your clinical knowledge but also you're basically being evaluated by your personality. Even if you have no interest in whatever specialty you are doing, you have to act enthusiastic at all times, act interested at all times, and basically be a suck up the whole time you're there. It's like you're constantly putting up a performance. I have been evaluated on the way I smile, how much I smile or not, my facial expressions, the tone of my voice, my personality, anything that you can think of. Usually my evaluations comment on how I need to completely change my personality (I am definitely more of an introvert so I tend to be on the quieter side and look things up by myself if I want to learn about a topic, which means that you will get poorer evaluations than someone who is super outgoing and asking questions all the time). You are being evaluated by your personality/performance at all times and again this score is vitals in determining whether you get honors or not. This is not only about getting a few letters of recommendations. This matters for every single rotation you do.
My being there contributes little to patient care so I rarely get the satisfaction of feeling like I did something useful.
Edit: forgot to add that you still have to take boards in MS3/MS4. Also MS4 is challenging because you have to be doing rotations (changing every month, putting on a performance to impress other people, studying for tests, taking boards) PLUS be working on your applications and traveling all over the country for residency interviews and being stressed about whether or not you'll match. Additionally, for some specialties you have to do audition interviews, which means you have to travel to another institution, be there for a month and basically work your ass off to impress everybody, and then do that several times over the years (specialty dependent but for some you need multiple aways).