First of all, I am an Allergist/Immunologist, and have been in practice for 2+ years. If you KNOW you eventually want to do A/I, it does not matter if you go in from Peds or IM, so just go with the one that interests you more. Fellowships for A/I are extremely competitive, so you are going to have to be a super star to get a spot. I think there are only like 80 programs in the country and most of them are small and only take 1 or 2 spots a year.
You have to cross-train on Peds and adults in an A/I fellowship. The A/I Boards are some of the hardest that you will ever find, because they are PACKED with basic immunology.
The field itself is challenging, even if you do private practice Allergy. It is not boring! I saw a post somewhere that said people don't die in the this field, well just an FYI, people do die from asthma (which we take care of), anaphylaxis from food allergy, drug allergy, etc, and there is a risk of anaphylaxis with the allergen desensitization therapy that we administer.