I started out in a family practice residency and decided to switch to psychiatry after the first year. From my experience, it's very difficult to switch. There are many issues: medicare funding rules, reflection on your character, programs needing to accomodate someone new, leaving more work for the people in your old program if they don't find someone to replace you, finding the time to interview, and getting LORs from the people you work with, including the PD and saying that you are leaving.
You definitely need to talk to your PD if you are leaving because the new program will need to call your old program and ask about your performance and academic standing.
I decided to start from scratch and applied to a first year position through eras. During my interviews, some programs flat out said they would not consider me because they will lose money accepting someone who already used up a year of medicare funding. Every program grilled me on my reasons for switching. In the end, it worked out for me as I will be happily starting psych residency near my home this coming july. But while most my medical school classmates will be attendings soon, I still have 4 more years of residency to go.