From what youve said it definitely sounds like you should get out of your Ph.D. program, however make sure to get that masters. Is medicine the right choice for you, only you can answer that so definitely get that clinical experience, and maybe take more than a year off before applying to med school, to decide that. You might be asked, If you quit grad school, what is to stop you from leaving an MD or DO program?, can you answer that appropriately? Or the question why medicine? Regardless of which area you go into, it is going to be a big commitment and require a major part of your attention for 4 years of school plus several in residency. Maybe explore some other options like social work, MPH, nursing, ect., if you like being in the area of healthcare, but arent fully sure that the MD path is for you. Since you mentioned physical therapy, you should try to get some volunteer experience in that area, which you should be able to find at your local hospital or medical center.
If you just sat in on physics classes do you think you would really learn, or just tend to show up irregularly, as for me I have to study for tests, attend labs ect. to really learn the material, but maybe just lecture works for you. In your year off you could consider getting a job at a University that would let you take a course for free every quarter (this is a typical benefit of most staff/faculty jobs at schools), and you could make some money and get physics out of the way. Your GPA is fine to get you in somewhere, just spend time really studying for the MCAT as it will improve your interview options. It might be beneficial for you to have those physics grades in before applying, to give the adcoms one less reason not to accept you.
A big problem is going to be not having your grad advisor as a LOR, as that is probably the most important person to get a letter from if you're going from Grad to Med, as some institutions require a letter from the grad advisor. I suppose if you have conflicts with them, you could definitely explain that if asked about it. I would just make sure to get a letter from someone in your grad program, committee member or instructor, as adcoms will want a current evaluation of your academic abilities. The best advice on this situation, can be attained by talking to the admission dean at some of the schools you are looking at attending, as most should be willing to sit down and discuss your situation with them. Just my .02, best of luck with whatever direction you take.