I worked as a hospitalist for 3 years after fellowship before going back for my child abuse fellowship. Pros: attending salary is way better, having a kid as an attending vs having a kid when you're a resident or fellow is way better, definitely gave me a better appreciation for "real-world" medicine as I was on my own in a private practice model. I had already passed the gen peds boards when I applied to fellowship so didn't have to worry about studying as a first year fellow, and I had plenty of actual experience when dealing with my sub-specialty and I think that helped. Child abuse is not a competitive field with only a few applicants per year, but I received an interview offer at every place I applied. I do not think that having 3 years between residency and fellowship harmed my application in any way.
Cons: It was hard going back to a trainee's salary, and it was SUPER hard to go back to not being the one in charge, especially at the end of fellowship when I had completed all my research and clinical obligations and was just marking time.