I do try to learn about math/physics in my free time, but that comes few and far between when I'm trying to focus on being a pre-med (which is very time consuming).
I feel like I'm selling out because medical school seems like the easy way out. I never feel challenged in a biology class. It's always just like "here are hundreds of facts and observations, know them all". In math class, my mind is constantly exercised and has to use reasoning and application of knowledge.
I feel you on this one! Although I fortunately/unfortunately majored in essentially what you are talking about
My advice, watch engaging theoretical physics videos and lectures online! It is far less time consuming than taking a course, and usually much easier to grasp than what classes will make you do (derivations and complicated problems). You won't be great at applying theory because you won't practice it much, but you will still learn the hows/whys/our best descriptions of the universe. This seems like what you are interested in doing anyway. Not to mention you can learn at your own pace and speed up, slow down, pause, fall asleep... do whatever you want. Here are some of my favorites:
Feynman's 4 part lecture on the basics of quantum electrodynamics (I have a feeling you will really enjoy this or pretty much any video with him in it. I feel most students that take basic physics would enjoy this actually, because he describes what we already learned in basic physics via QED)
http://www.youtube.com/watch?v=eLQ2atfqk2c
http://www.youtube.com/watch?v=kMSgE62S6oo
http://www.youtube.com/watch?v=jNNXD7fuE5E
http://www.youtube.com/watch?v=UigjOJm6F9o
Paul Callaghan explains MRI and NMR in 10 lectures:
http://www.youtube.com/watch?v=7aRKAXD4dAg
Stanford's modern physics class on youtube:
http://www.youtube.com/watch?v=oWe9brUwO0Q
Stanford's General Relativity intro:
http://www.youtube.com/watch?v=JRZgW1YjCKk&list=SPpGHT1n4-mAvcXwzOIz3dHnGZaQP1LEib&index=10
Entire Nuclear Physics Fundamentals and Application course (thick indian accent but good material):
http://www.youtube.com/course?list=EC907E24DB6A5CD46E
And screw it, here are all the school's youtube playlists that have awesome stuff for freaking FREE:
MIT:
http://www.youtube.com/user/MIT/videos?flow=grid&view=1
UCBerk:
http://www.youtube.com/user/UCBerkeley/videos?flow=grid&view=1
Stanford:
http://www.youtube.com/user/StanfordUniversity/videos?flow=grid&view=1
There is so much out there to learn for free. I haven't really gotten into mathematical videos because, unlike physics, I got sick of complex math by the end of college. For me, math tends to get more and more specific as you learn it (like building more refined tools) whereas physics still keeps a big picture. That is why I got bored of math
Don't feel like your selling out, you have a lot of life left to learn. Imagine how hard it was to learn this stuff 30 years ago. The time it would take, and the limited resources to learn modern physics then would be such a large barrier. Now you can do this 100x faster and more efficiently...
Hope this helps you procrastinate!!! The second link I sent is a good one to fall asleep to.