I am a little surprised that you are taking anecdotal information from a single source so seriously, without having investigated, for yourself, his claims. You are a physician, you are probably smart enough to use pubmed or even google to find out for yourself what the facts are.
Can your source cite a study that backs his claims?
Nothing is risk free, but if you maintain proper precautions ALARA principles and radiation safety your risk will be significantly be reduced, such that it is not significantly greater than that of non IR radiologists or other nonrads physicians.
If we are fortunate enough, we can choose for ourselves the life we want to live. It seems you have a choice. Do your homework, look up the facts and the evidence, decide for yourself. If the fear of radiation induced cancer is crippling to you, and you think the risks are not acceptable, choose another field.
Best...