Not at all. My situation somewhat mirrors your own: I am in the Navy, having received a civilian residency deferment (radiology). I also applied for and was/am deferred for a civilian fellowship. I did not go on a utilization tour and my fellowship immediately follows my residency.
The fellowship I was deferred for has never been granted before in the Navy. Right place right time.
You can maximize your chances of getting a fellowship deferal by talking with your specialty leader and honestly discussing your desires/situation, especially if the fellowship you desire might not be a high-priority for the Navy. My situation is the exception to the rule because I have basically gotten everything I have desired so far from the Navy. *furiously knocking on wood*
Remember, there is always a chance... (reminds me of a quote from Dumb and Dumber).
I would recommend applying for the Navy pre-selection board if you are far enough out (2 years from fellowship start time). If you don't get selected you can always apply the following year for the selection board and have the potential to scramble into a fellowship program (not ideal, but possible).
Good luck.