If time is such a concern to you, why are you willing to complete a rigorous three year fellowship in cardiology to only do practice in this field fifty days a year?
In line with what Kimberli alluded to, if my family member told me they were going to see a cardiologist (or any specialist, for that matter) that only practiced one day a week, I'd have serious reservations about the care they provided and would suggest they seek care elsewhere. I just don't think you can stay current and at the top of your game only seeing patients one day a week--unlike semi-retired docs, it's not like you have been working for years and have lots of experience to rely upon, you NEED to gain this experience to be a competent cardiologist. Plus, how will you be available to your cardiology patients during emergencies when you are wearing your "FP" hat? Your overextension and broad practice base may significantly limit your availability to your patients, and lead to sub-par care.
I admire your commitment to your community and willingness to help those who truly need your help, but I don't think your plan is logistically feasible, and could potentially be unsafe for your patients. I'd love to see you prove me wrong, and do all that you want to do and DO IT WELL, but I don't think it is feasible. Med-Peds residents seem to have similar goals as you have, and are eligible to complete any fellowship in medicine or peds, and would likely give you the most flexibility. I'd recommend trying to do one thing really well rather than trying to fill in all the holes in your medical community.
Best of luck, whatever you decide!