Use of antibiotics and other medications would probably be a huge issue. As much as we would like to believe that judicious use of indicated antibiotics is a priority, the truth is a ton of primary docs of all stripes (IM, FM, Peds) Rx them because it's convenient and is what the patients want. You may believe that this causes more harm than good, but no matter how hard or well you explain that to the parents who want an Rx for their kid's cold, your partners are probably going to resent you when your patients are rescheduling with them because you didn't practice in a manner consistent with the longstanding patterns of the practice and the parents were not happy with your (perfectly reasonable) care. While the antibiotics are probably a more "global" issue, it goes just as much for simethecone, cough syrup, liquid albuterol, and probably a good few other useless medications that are routinely prescribed.
Likewise, if you work in a practice that likes to do major portions of their own workups before sending to the subspecialist and you feel more comfortable punting early (or vice versa, which I'm sure is getting more common these days), that could also be problematic.
Do you feel comfortable managing complex kiddos or would you rather bounce those kids the group at TBCH if reasonably nearby? I can tell you the names of the best pediatricians and groups on my side of the state and I can tell you the names of a few that...well, aren't. Groups of people comfortable with the complex kids probably congregate together and appreciate their immediate colleagues' ability to handle the same.