Would you give up pre-med?

You need to be honest with yourself and your prognosis. When you say it will get better in time, is that time soon enough to allow you to complete all your pre med coursework and ECs? It's not a good sign that you had to withdrawal so many times because of your health. You can't do that during med school, and med school will be much more rigorous than undergrad. You still have time to think this through and see how well your disease can be managed now and in the future.

Is medicine the only career path you could see yourself pursuing? Once you apply to med schools and explain why your schoolwork has suffered, adcoms are going to want to know your disease is managed enough to make it through med school. So not only do you need to convince yourself you can do this, but you also need to convince other people.