I suppose that's a personal question as to "worth". If you really enjoy working with both, do you feel like you would miss out if you only took care of one versus the other? Additionally, are you positive that in 4+ years that you will want to specialize? If you have uncertainty to these questions, then maybe it is worth it.
That all being said, if you know you want to do a fellowship, at this point in time, it is doesn't make financial sense to spend an extra year in residency to learn about patients you won't treat again. I don't have a lot of knowledge about combined fellowships (they are not regulated by the ACGME and are offered only by specific programs or in specific situations), but I do know they exist, but generally for only a handful of specialties (eg, I've seen it for Nephrology, Rheumatology and Allergy/Immunology. I suppose you could put Adult Congenital Heart Disease in that group, but you don't need med/peds for that track. I do know someone who also did a combined cardiology med/peds fellowship after a med/peds residency, I think his salary end up capping at PGY8 but he was in training for a decade). It doesn't really make sense why it isn't more commonplace for those who want to pursue it (knowing the significant time investment), given the fact that most childhood diseases end up being future adult diseases and in general, the transition of care from pediatric specialist to adult specialist is often mediocre at best, but that's the current way. Your observation are in line with what I've observed over my career though, in that most med/peds people who pursue subspecialty training end up practicing either IM or peds. Additionally, they tend to give up their board certification in the specialty they don't practice, mainly because it is not worth the time or money to maintain certification. Maybe someday the ACGME will create a track for med/peds people who want combined fellowships, but I don't see it happening in the foreseeable future.