You might not get a chance to pursue that fellowship at a later time depending on which one it is and how influential the place is that you are backing out of. Also, make sure you didn't sign a contract that requires you to show up or face financial or other penalties...
Largely depends on geography, how big of a world your specialty is, and how difficult it will be for the program to find a replacement.
Smaller cities and smaller specialty communities make it more likely that the program will be able to bite you with any leftover ill-will. The closer to the start date and the presence of a match make it more difficult for the program to fill your spot, making them more likely to be upset with you. And, of course, if there is a match, then you're probably burning that bridge vis-a-vis the matching organization.
Legally, my understanding is that the only they go after you for are the expenses related to your recruitment and hiring. Few people bother to do this, as it is not typically enough money to make it worthwhile.