To leave one program for another though, usually requires a "letter of recommendation" from the previous program director. You have to leave in good faith and be "in good standing," whatever the heck that means. I don't know that this is a universal requirement, but we've had some residents leave our institution mid residency, and it was really quite awkward.
The PD really holds the keys, sometimes.