Reasons that people have told me that attendings have it harder than residents:
1) Work hours. There are no work hour restrictions for attendings. There are also no work hour restrictions for florists, but they don't work 80 hours per week either. Attendings don't work resident hours because they quit jobs that make them work resident hours. The only attendingss I know who routinely work more than residents are in short (1-2 year) partnership tracks with sweat equity. They can work 100 hours per week or more, but they know what they're doing it for and if they want they can walk away if they decide that its not worth it after all. Finally there aren't really any work hours for residents in the first place. Most residents I knew broke hours, and a lot of us weren't even close.
2) Back up. As a resident you have your attendings to come running, as an attending you have no one. Except... bull****. In most jobs as an attending you have attending colleagues that support you and consultants for the really hard questions. As a resident, when you ask your attendings a question they have every incentive to make you absolutely miserable to make sure you don't ask them any more questions in the future. As long as you don't take a truly isolated job (i.e. the only ED or OB doc in the hospital) you will probably have more support as an attending than as a resident.
3) Guilt. As a resident you aren't really responsible for your patients, and attendings are. Except that any resident who is even halfway decent human being owns his patients emotionally and ethically. Which is fine if your attending is also good, but when they're shoddy or stupid you now feel ethically responsible for bad a situation you can't control.
4) Risk. Attendings can get sued. Except that in a lawsuit you have the right to a defense, and in almost all cases even if you lose the payment comes from your insurance rather than you. Residents can be extended, which costs them hundreds of thousands of their own money and is career maiming, or just fired, which is career ending. And if you're facing discipline as a resident your accuser is also your judge and jury. I never really felt safe until I was out of residency.
My experience is that attendings who complain about being attendings are usually doing it in the context of responding to a resident's complaint or concern. I have never heard any of my non-academic colleagues wish that they were back in residency.