I'm the OP.
I stayed in ortho, I'm glad I did. Almost a PGY4 senior resident now, and have published in major journals, presented in a bunch of national conferences and will be applying to fellowships next year, job offers are already coming in. The lesson here is don't make rash decisions during the abyss of your ortho ED consult rotation on your 2nd 30-hr shift of the week in the middle of winter. Wait until your rotation gets a little more humane, the sun a little warmer and longer to make a decision. Residency is not forever, and the more senior you are, the more you delegate and appreciate the intellectual aspects of the field. I also shadowed an ED resident for a bit on the excuse of learning to do some ultrasounds to get a different perspective of the ED and I think that helped reduce my grass is greener itch, namely undifferentiated patients are sometimes more an exercise in frustration rather than an intellectual challenge to be solved, especially when you have a full waiting room of patients with vague abdominal and pelvic pain.
Some of the original issues I brought up still hold true in many instances, but I've also come to appreciate a lot of things about ortho that I didn't get enough exposure to as a PGY1 or PGY2, such as the patient relationships you develop (patients do remember you years from now) and also a sense of "craftsmanship" in the surgeries you do. Surgeons are definitely not all equal, and even routine surgeries like total joints are never really that routine (it's not just the same 3 surgeries); there is great deal of satisfaction in doing better, achieving better outcomes and avoiding avoidable complications... in being "the expert." The shift work in the ED is enviable, but the twice daily signouts are not. The further up you climb in ortho, the more tolerable the hours and the better the ancillary support. Most of the ortho attendings I know (other than the trauma guys) in academics and community rarely work nights. The hours are longer in ortho, but more regular and conducive to family life. Now that my wife also works a more regular schedule, that is a distinctive advantage once out of residency.
Don't make rash decisions under stress, trust that the decision you made as a medical student was done with a proper inventory of your interests, skills and ultimate fit, even if parts of residency itself may not correspond well to them.