I switched. It was crazy and stressful but it all worked out in the end. I started a Peds residency but after a month or 2 I accepted what I already knew, I wanted to do PMR. I finished the year (yes it counts as my "prelim": the requirements for most, or at the very least PMR, is "6 months of Medicine, Peds, FP, or Surgery in an accredited residency"), rejoined the match, and was fortunate to match into a great program.
To clarify the above, I am 99% certain that the rules from the NRMP are this: your matching agreement is to "sign a contract at your program". After you do this, and I suppose work a day or two, you are "free" to quit. Obviously this looks bad, really bad actually, but I do not believe that you would face any NRMP repercussions (PLEASE call the NRMP to confirm this; they are very helpful on the phone). However, DO NOT QUIT mid year unless the circumstances absolutely demand it.
Some things I learned in the process:
a) Finish the year. I think this is important, though I know it is hard to be somewhere you don't really want to be, but quitting looks really bad. I'm sure it happens, and people go on to have fine careers, but I would not advise it. While on interviews in my new field, I was actually complimented several times by interviewers for 'sticking it out'.
b) Do it the right way. Start telling people from the top down, start with your PD, then your Assist PD, then Chief Residents. Definitely be positive. Next personally tell as many attendings as is reasonably possible. You'll have to work with them the rest of the year......better they hear it from you. Then tell your class, of course you can tell people you trust first, but this is big news and tends to spread fast. Everyone will respect your decision, even if they don't like it, as long as you are sincere. Your program director can be a big help in the re-applying process. Mine wrote a letter of recommendation for me and this helped tremendously.
c) No one knows how the funding stuff works. I still don't. Not one PD or interviewer ever mentioned it. What I understand to be the case is that you are 'locked' into x number of years of funding when you match the first time. I was however, told by a chief of GME that you might be re-classified after matching a 2nd time.....I have no idea if this is true. Anyway, you do not have any control over this so, try not to worry about it.
Good luck......PM me if you have any other questions.