Most epi programs prefer that you've taken calculus, but this tends not to be a hard-and-fast rule; it depends a bit on your interests and what other strengths/background you bring to the program. If you're thinking of dropping it now, I'd recommend staying with it unless it's too excruciatingly painful. But you don't need to kill yourself taking advanced calculus/differential equations/etc. if that's not really your thing.
There is a *lot* of statistics used in epidemiology, but you can get away without having taken calculus in most basic first-year biostatistics sequences. Calculus just doesn't get used there, in most cases. Many epidemiological concepts are grounded in math, but unless you're interested in theory/modeling/methods, you probably won't have to understand things at quite that advanced a level; you'll just need to know how the concepts/techniques are used and why they're important considerations.
Calculus is certainly useful for more advanced classes in theory, modeling, advanced statistics, etc., but you will probably find enough other elective classes to take that you will not be forced to take any classes that require calculus. Just know that there will be certain things you won't be able to do without calculus. Which are probably not things you have any interest in doing, anyway...
Hope this helps!