Mileage may vary program to program. I don't think it would be a case of non-clinical letters being held against you--they just may not give the programs the type of information about you they are looking for.
Personally, I like clinical letters from multiple specialties because I am trying to assess the strength of your performance in that arena and your ability to work with the healthcare team. I like letters that tell me that you had great rapport with families or that you managed patients well and participated on rounds, or that you were dependable and trustworthy and prompt with your notes, etc. The non-clinical letters don't seem to tell me as much and generally leave me with a feeling of "that's nice," rather than a "Wow! I'd want to work with that person."
But you ultimately have to go with what makes you feel most comfortable and confident. If you think the non-clinical letters allow you to be seen in the best light, you should probably go with those.