I believe there is a very simple answer to your question. You should definitely begin pre-med IF you KNOW that you want to be a physician with all your heart and you have a CLEAR understanding of the sacrifices that one needs to make to become a physician, and thereafter. If you can answer yes to the above, then you are in a perfect place to begin your medical career.
I am 26 and I'm hoping to get into med school this year (3rd try). Assuming I start med school this year, I'll be 27 by then. In four years I'll have my MD at the age of 31. I know that I want to be a surgeon, so I have to go through at least a five year residency. I'll be 36. I'll probably want to specialize, therefore, I'll have to do a fellowship. I'll be around 38-40 years of age by the time I'm a full fledged praciticing surgeon. Now let's assume I had started right after college, 4 years ago. I would still be 36 years old before practicing. What I'm trying to say is the age issue is NOT an issue. Actually, many schools prefer older, hopefully wiser and maturer, applicants.
In addition, if you already have an undergraduate degree all you have to do is to take the required pre-med courses. There is no degree called pre-med. It is just a bunch of requirement that one has to meet. If you go full time, you can have them out of the way in 2 years (this is what I would recommend). Plus, you will have a clean slate. Therefore, if you are committed and thus earn good grades, you can have a very high GPA. After that all you have to do is bust out the MCAT, apply and your on your way. You may not want to do a long residency like surgery, so you can be practicing by your late 30's.
In summary, you are in a position of advantage because you have a clean slate to work with. You don't have to worry about your past grades messing up your GPA. If you really want to be a physician, then by all means do it. When it comes to the application process, the most important thing for you to do is to explain why you choose to pursue medicine vs. what you had studied earlier. If your answer is honest, there is no reason why they wouldn't consider you as a strong applicant.
In terms of the loans, that is just one of the many sacrifices a person must make to become a physician. Yes, you will have a lot of loans, considerably less if you go to your state school, but you will also make a MINIMUM of 120,000 a year as a physician. So what if it takes a while to pay them back. You can still buy a BMW
You seem to have many doubts, if I read your post correctly. This is the only thing that concerns me. To be successful in this road, you must be dedicated and sure of your decision before embarking on it. I believe this is the question you need to answer first and for most.
I hope all of my rambling helped.
Me!