It depends on the fellowship.
Breast fellowship is an exception because of relative shortage of breast imagers which will change in the near future. Otherwise, most groups won't hire a non-fellowship trained radiologist. It is necessarily not about the skills, it is also for marketing purposes.
A non-fellowship trained radiologist may be as good as a neuroradiologist but he/she has to spend twice as much energy and time to prove himself/herself to neurosurgeons and neurologists. In fact, many groups may not let you interpret the images that are ordered by neurosurgeons or neurologist if you are not fellowship trained.
Example: In our group all the MSK studies ordered by orthopods and rheumatologists and all MSK MRI is read by fellowship trained rads. Similarly, most neuro studies ordered by neurosurgeons and neurologists are read by fellowship trained people.
In summary, if the fellowship is IR, Neuro or MSK it will make difference even 10 years down the road if you want to change jobs. For academics also you need to be fellowship trained. The benefit of the fellowship in the long run is less for mammo, body, chest and Nucs.
One year of fellowship is no big deal and in most places you can do moonlighting and make more money and gain some experience. Taking into account the benefits that you can gain by doing one year of fellowship, I say it is worth it.
I still don't understand why people make such a big deal about doing one year of fellowship.