On the first day of physics, my professor "tactfully" told students that some people just "get it," and some people don't. "There are some of you who will study for 10 or 12 hours a week, and still won't get an A. There are some of you who will probably feel like class time is a waste, and you will get an A with no problem." I felt like physics was much easier than any of my other science classes and spent a relatively small amount of time getting As. (and as a result I was a TA in the physics homework lab for a year, much easier than when I did research).
On the flip side, I spent no less than 20 hours a week when I took my O-chem lab to barely pull out an A. (I did take it during summer term however, so I completed it in 7 weeks) MUCH more time intensive class, and required MUCH more effort on my part.
I would say that Physics is either in you or it isn't, and the time spent studying it has a small impact on one's overall understanding of it. O Chem is "difficult," but much more value, improvement, and understand can come from the time you spend studying it. It's not intuitive, there is a lot of memorization, and it requires a lot of time to "get it." The method for solving physics problems seems intuitive for me, but for those who don't find it that way, studying for extended periods of time can seem futile.