That's a strange question. But I guess if I had to chose, I would say Organic only because it's an entirely new language. Before you can actually learn more advanced material, you need to get comfortable with the terminology. Once you've established that, understanding additional concepts become more comprehensible.
And while many people might disagree, I think some degree of memorization is involved in Organic. While being able to understand and predict mechanisms is important, at the end of the day, you're probably better off just memorizing the main details: the reactants, reagents, and products. That alone will be enough to help you answer a vast majority of questions.
The topics presented in Physics and General Chemistry aren't really that difficult either. Much of what you learn is simplified into a sheet of equations. But, it's how those topics are presented (in the most obscure ways) and knowing when to use the appropriate equations that makes those topics challenging.
Just my 0.02c