Its not uncommon - we used to lose someone every few years to Anesthesiology or Rads. But as noted above:
there has to be a spot for you
your current performance has to be good
your current PD has to support your choice
Don't burn any bridges in case it doesn't work.