This a bit anecdotal but....
A friend of mine, currently a CA2, went from internal medicine to anesthesia without losing any time. The internal medicine program wasn't thrilled, but in the end, they'd rather have him leave than stay there unhappy. They even offered to help him get a spot in their anesthesia department, but it was a geographic issue. Having done an anesthesia elective in med school, he called up the program director and got an anesthesia spot at his alma mater.
I have to say my friend is infinitely happier. His only regret is that he had to settle for a second tier anesthesia program. Being AOA with a nearly perfect board score, he could have matched anywhere he wanted. So, you might have to lower your standards a little.... or you might get lucky, too. Even the best anesthesia programs go unfilled or have people leave.
Best of luck!