For Rutgers program, there is a significant cost to the host company (money is paid to Rutgers up front, and there are very specific dates for start and for the end, when the one-year and when the two-year fellow may accept a job offer and officially leave the program). My guess is that most other programs have similar arrangements. Depending on the size and reach of the program, you are likely going to burn bridges with that company, that program, and those who are connected with them. Is it the death sentence to one's career? No, but it will limit your options. Because I constantly keep running into people I used to know 10-15 years ago... and people talk about people they knew. Usually it's either those who are exceptionally good or exceptionally bad, mediocrities tend to be forgotten - but something like that would be pretty significant.
So in terms of consequences - no, they won't be taking your to court or something like that (though if they paid for your relocation, they would likely ask for that money back), but in terms of acquiring a certain reputation that may travel far and wide (depends on the reach of the program) - yes.
If it's not an industry fellowship but a clinical one with one of the organizations focusing on very specific therapy area - I don't know how those programs work and can't comment.
Since the devil is in the details - only you can make that decision. Ultimately if you see your career long-term in the field where you are doing your fellowship - don't burn the bridges, they will come bite you in the back. If you will be perfectly happy with the types of jobs like the one where you got an offer - go for it and be prepared to answer if someone asks (given that you just started your fellowship, I would leave it off the CV, then someone would only ask if they recognized your name and your story), exactly why leaving was the best decision for you, and why you aren't really a job hopper (because no one wants to hire someone only to have to go through the hiring process again 2 months later, so if your job history shows you are leaving for greener pastures every few months, that will count against you).