Bad idea.
If you want to be sure, buy a full practice test and take it under testing conditions (in an empty room, same breaks, no internet or notes). If your score is acceptable without taking physics than it's okay.
You will still need to take physics before beginning medical school.