Coming from a 1-4 program where over a third of my class did fellowships, exposure during residency was a reason for choosing to do a fellowship. They found a field they loved and the easiest way to carve a niche is usually a fellowship.
The difference in income for that extra year and then a fellowship versus attending straight out of a 1-3 is substantial. However, unless you have a small army of children with a stay at home spouse or a crippling gambling habit, the pay with moonlighting during fellowship is going to be more than enough to be making payments on your loans and enjoy a small-medium bump in standard of living from residency. And after living like a resident (essentially middle class) for four years, one more year probably won't feel like a deal breaker.