I am a physics major. I have had many people say to me that adcoms won't "like" physics b/c it isn't related to medicine. Of course, they are naive.
The point is, physics isn't hard at all. For me, anyway. It looks like you enjoy physics, so you are probably like me, mathematical and analytical in nature. Physics isn't hard, if you know what you are doing. In fact, physics is easier than biology and biochem for ME. I don't have to work as hard and I can achieve really high grades. My physics and math GPA up through my sophomore year (I'm a junior now) is 3.93, spoiled only by an A- in partial diff eqns. Also, physics tends to have small class sizes, so you can know your profs and get them to write you a good reference.
Are the classes difficult? That depends on the person. Quantum mechanics is the most challenging course this year for me, but it's also the course that I'm getting the highest mark in, going into the final. I gather it's because I enjoy it so much. If you combine your degree with math, you probably have to take courses like Real Analysis (which is an abstract course, focusing on proofs and set theory) and topology (topology deals with the geometry of sets) which are very difficult, but do-able. Next year I will take graduate courses in quantum mech, quantum field theory and perhaps classical E+M on the level of Jackson. Grad courses have an A average... so I expect to get A's in my graduate courses. One of the most difficult courses I will encounter, I am sure, is statistical mechanics next year. I know there are people complaining about physical chem and how hard it is, but stat mech is a hundred times harder than physical chem... Remember, in chemistry they apply the formulas. In physics, we PROVE them. And in general, proving things requires a lot more ingenuity than just being able to perform calculations on cue.
So ask yourself these things: Do I like to find out how things work on a fundamental level? Do I like using precise mathematical statements to describe nature? Do I want a mathematically rigorous and demanding curriculum? Or am I satisfied with the qualitative, descriptive nature of a "soft" science like biology? If you answered yes to the first three and no to the last one, then physics is probably the major for you. I made that choice, and I haven't regretted it once... in fact, it's made me feel a lot more secure, knowing that I will get a degree in something I enjoy, and that I can go on to grad school, if my med school plans don't work out.