NOTE: In my IM centric world, I assumed this was an IM fellowship. Looking at your post history, I see that you're in psych. So some/all of this may be wrong. But I already typed it out, so you're stuck with the advice.
It depends upon the details and how you define "damaging".
If you no longer plan to do the fellowship for personal reasons, you could apply for a waiver. If approved, then there's no long term problem -- you're released from your fellowship and can participate in the match again in the future if you wish.
If you don't get a waiver, then you'll be in violation of your match commitment. The only price you'll pay is inability to use the match again for a few years. This would make it very dfficult to get a spot in most fellowships. You would still be able to get an off cycle spot outside of the match. Fellowships might be less inclined to consider you since you dumped one in the past, but with a good explanation I expect many would be fine.
This shouldn't affect your ability to get board certified, nor get a job. Except at the place where your fellowship was based -- they might be upset. But again, with a good explanation, they may be totally supportive.
