NoktorNoL, who did you have for quantum? I had Pate and he was AMAZING. I had Lehmann for thermo, though, and let's just say it was a less-than-stellar experience... I think the average on one of our exams last year was about 20%. (That being said, the final grade had a ridiculous curve, and most people ended up doing fine.)
To the OP, it helps to remember some calculus, but it's more important that you're able to understand what's going on when you use certain equations, you'll be alright. I took multivar and linear algebra in high school, and I didn't remember a thing of it for pchem. Maybe it would have helped, but it wasn't crucial. A lot of people prefer pchem to biochem, actually, because it requires less memorization and more understanding of abstract concepts. Some people just work better that way.
A lot of it also depends on the professor. I think pchem is really only difficult if a professor doesn't know how to teach it to the students in a simplified manner first and work up from there. Both of the professors I mentioned above at my school are absolutely brilliant and probably at the top of their field, but only one of them knows how to communicate with students at all. I think most pchem professors know it's not generally a favorite class for most students and they try to make it at least mildly intelligible. That, or provide it with a generous curve.
Personally, I loved my quantum semester. It's really mind-blowing how much hand-waving we do in organic chemistry. My favorite pchem moment was when I was talking to my professor and he was explaining how p orbitals are actually donut-shaped and not dumbbell-shaped as we're always taught. He said, "I bet it's one of those things that some of my colleagues probably don't even remember." And I responded, jokingly, "Yes, well, it's probably not the sort of things that keeps people up at night." To which he replied, with a straight face and a slight note of frustration, "It keeps me up at night."