If you withdraw after matching, then you have committed a match violation, unless you received a waiver from NRMP. Being barred from the match for 3 years, yes, would make you considerably less competitive.
If you withdraw after interviewing but before matching, it might be prudent to contact programs ahead of time, especially if you corresponded with fellowship directors and started on the pre-match dance of admitting interest without violating the match. Just to let them know why you withdrew and that you're still interested in their program.