I'd say the physics is a little harder because the mathematical arguments used in calc-based physics -- especially E&M -- are more rigorous than people are generally accustomed to doing, unless you've had a lot of calculus before. Orgo 1, generally, isn't too difficult, but requires a lot of work. Physics E&M with calculus uses a lot of vector calculus for explanation, not necessarily for practical usage on exams though. I did have do vector/multivariable calculus on homework sometimes, but exams didn't necessarily test that knowledge specifically. However, I do feel that the extra math intensity allowed me to understand E&M pretty well.
Also, if you're math savvy, you'll probably like E&M a lot.