What apumic said. I'm a math major and there are people who are excellent at calculus, algebra, etc. who struggle bad with the abstract stuff you have to take. If you have trouble with regular stuff, you'll be miserable.
If you think you can solve your issues, by all means, try. Math is awesome, but it does build upon itself, and trust me, you won't like it at all if you don't get the basics.
Why don't you just enroll in some basic math classes and see how they go? If you struggle, I'd try and find something else, but if you can get through it, enroll in a math major course and see if you still find it interesting. Then go from there. You'd be surprised how surprising these courses are. It's like relearning math all over, but in a harder way. Def. not what I was expecting from my major, but I still liked it.