Considering Calc Based physics is not required for medical school, I can't imagine it would be problematic. Do your due diligence with your school's Health Professions Office to make sure, but I wouldn't lose much sleep over it.
David D, MD - USMLE and MCAT Tutor
Med School Tutors