Well in regards to getting a fellowship, I don't think it helps much IMHO. Research, strong LORs, networking, and in some rare cases in service scores, trumps being a chief by far. Believe it or not, some places still look at Step scores still. I actually had two rejections because they said my board scores did not meet their criteria. So in regards to being chief for fellowship, I don't think it matters really. Sounds like from the above posts it doesn't really help towards landing a job. So I guess from a strictly objective and practical point of view there isn't much to gain by being a chief and potentially just more hassle to deal with your final year of residency in which you could be chilling, studying, rounding out your knowledge and finishing up any research.
But for whatever its worth I think it can help in the future on your practice website stating you were a chief resident and it shows you were a leader amongst your peers I guess. Personally, I don't think I can stand doing things like making a schedule for other people and settling scores between faculty and residents. I have enough trouble organizing my own life. And I have a short tolerance for inefficiency. But I think if you're super organized and passionate about the program I definitely think one should do it because it can't really hurt and there are a lot of intangibles skills you gain from it like conflict management and organizational skills...which may pay dividends in the future. I used to think being chief would be a waste of time and energy but in retrospect I think it is a good thing to do for its value beyond the CV.