That's a really hard question to answer. For some people, it's the hours. I just got off a 30-hour shift. For some people, it's being around sick, and often dying kids. For others, it's dealing with family members who make unreasonable demands or have expectations that just aren't realistic given the disease process and/or your resources.
For me, it's the fact that residency takes away from other parts of my life. I don't hike, camp, cook, read, or write nearly as much as I used to. I've lost friends along the way because I didn't make it to a birthday part, wedding, or other event. Peds is nice because depending on what field you choose, you can easily have a life outside of work. An almost normal one at that. But almost without exception, your outside life will be severely limited in residency. It does, however, get better.