Physics does involve quite a bit of math, but for Physics 1 and 2 at least, its not really too bad. So if you've felt comfortable with the math courses you've taken thus far, you should be fine. However, if you've never been too much of a math person, then maybe it would be a good idea to take some math classes and make sure you understand how to do everything. In physics, the professor will pretty much assume that you already know how to do the math, so if you're not to proficient in it, you may need to work a bit harder at it than others.
Best