Addendum: One of the most poisonous things you can do is trick yourself into thinking that the department is like, the regular thing that you do and you'll miss out on stuff when you're not there because you don't see the regulars and won't catch up with them and all that.
It's a shame, because I work with some pretty great people, and yeah, I miss them when I'm gone for an extended bit of time.
In a good health system, with reasonable staffing and expectations - this would go a long way to career longevity.
In pro sports, there's a term called the "glue guy" in the locker room. This is the guy that keeps morale up and has unique ways of generating team chemistry. He's the glue that "sticks the team together". He's not a superstar. He's not even an all-star. But damn, are the superstars and all-stars happy when he's around. For those that follow hockey, the Pittsburgh Penguins have three superstars in Sidney Crosby, Evgeni Malkin, and Kris Letang. The glue guy during their cup run was Max Talbot. Nobody recognizes his name. Not a goal-scorer, not a playmaker, not a sniper, but he kept the team heads-up-and-happy on their feet.
I like to think that I fill this role in at least some part at my current shop, and I have reason to think I'm not wrong. I'm sure not some procedural wizard or brilliant diagnostician.
But man, the more that you're in the department... the more gamma rays of patients, admins, and all the other nonsense you're also exposed to. I'd be more of a glue guy if it weren't for that.
The ED ain't your hangout. Oh, no. That's a baaad trap to step into.