I was a double-major in Computer Science and Mathematics for my first two years of college. I then added biology as a third major, but I took about 12 of the biology major courses during the summer. The difficulty of computer science courses can become more and more difficult or more and more easy, depending on the student.
Let me explain. There are two types of students in upper-division computer science classes: Those who mastered coding in the previous two years, and those who didn't. The latter group suffers from the difficulty of upper-level programming assignments combined with the difficulty of learning a language that they should already know. This latter group dramatically brings down the curve and makes it a breeze for the former group who spend most of their time implementing labs without having to learn the language.
At UC Irvine the course that most exemplified this dichotomy was Computer Graphics, which required students to use C++ for the first time while working on nasty OpenGL projects. UCI students learn Java during their first year and have little exposure to C++. I had a much easier time due to some research programming I completed in C++ that taught me quite a bit about the language.
I advise you to assess where you stand with your peers. Are you generally more competent at programming than they are? If so, stick it out! You will find that your mathematics knowledge (especially linear algebra) will become insanely beneficial in later computer science courses.