I don't mean to be a downer, but you really need to come to terms with the notion that math is important & relevant in more fields than you may think. Unfortunately, we live in a society where it is commonly accepted that math is "icky" and irrelevant. What I mean to say is, of all the subjects taught in school, the one which it is most OK to be sub-par in seems to be math. This is not just my own perception; a professor who taught me differential equations would regularly express his frustrations regarding this very matter (not that 2 people proves anything...).
To be able to perform well in mathematics has far greater implications than just being a "math wiz." Think about what separates math from other subjects. You are required to learn basic principles, as a previous poster mentioned, and you are then expected to apply them to new situations. Math aims to test your problem solving ability in the purest, most basic sense. Does this sound at least somewhat applicable to the field of medicine? I think so. Moreover, the idea that you will never encounter math as a physician is, in my opinion, an understatement. While methods of differentiation and integration (the fundamentals of calculus) may not be common fodder amongst physicians, statistics (e.g. Student's t-test, ANOVA, and other analyses) will be of great use if you decide to engage in research activities or even interpret journal articles relevant to your field.
Additionally, topics taught in calculus are DEFINITELY put to use by engineers in their clinical research (yes, engineers DO perform clinical research, too). If you find yourself collaborating with a group of biomedical engineers, you may have to brush up on your calculus to understand, for example, the mathematical modeling currently being put to use to better understand physiological systems.
I realize that these examples may not apply to the majority of people entering the field of medicine, but to deny the significance of mathematics because you "want to be a doctor, not an actuary" is just sidestepping the more important issue. Math and math-intensive courses, such as physics, challenge you in a way which other med. school pre-req.'s do not. Most importantly, anyone can become proficient in "math-y" subjects by putting forth the necessary effort. You just need to stay on top of things.
Make a plan to do a certain number of problems each night. Most students who take a wrong turn in a math/physics course, do so because they allow themselves to get behind on the problems. I cannot stress this enough: PRACTICE PROBLEMS ARE THE MOST EFFECTIVE WAY TO IMPROVE YOUR GRASP OF THE MATERIAL. Once you are behind, you are going to stay behind. These things tend to have a snowballing effect. Make use of your school's resources, such as office hours and tutoring, if necessary. If you approach the subject matter with a can-do attitude, you will definitely be able to master the material. And that is what you should aim for: mastery. Own the problems, own the tests, and you might even find yourself enjoying a subject you used to dread, not to mention you can T.A. the class later and make some $$. Sorry to impose on this thread with such a long post, but please don't brush off math/physics because they may not seem important. Once you have mastered them, you will see their effect on your overall critical thinking abilities.
Not to be mean, but rote memorization (i.e. memorize & regurgitate), while useful in the pre-med curriculum and even during the first half of medical school, will not be as helpful when you are trying to diagnose/treat a patient (heck, you can always brush up on details with a manual or your Palm Pilot anyway). What is really going to make you shine, in my opinion, is your problem solving ability. Math and physics go a long way in developing this faculty, so please challenge yourself with them.