Your concerns are completely valid. Student loan debt is not good. Realize that you will have debt equivalant to many people's mortgages, sometimes a mortgage on a very nice house, depending on where you live.
I would disagree that being a doctor is worth it (I used to think it is worth it, but now I wish I hadn't gone to med school) A rapidly increasing number of physicans share my thoughts. Many of us are burned out but feel trapped becuase of the student loan debt.
In addition, many specialties salaries have decreased dramatically, with many primary care starting at just above 100K. Some pediatricans start at 90K. So it's entirely possible that you will have a hard time making even the minimum payments, much less paying it off early. Add to that the fact that as far is the governement is concerned, if you are a doctor you are "rich", therefore evil, and therefore deserve to be punished by taxes. At a physican's income, you will not be able to deduct your interest paid.
Plus the loan companies play game to make it as confusing as possible to keep up with your accounts. They do stupid stuff like create a new account for each dispursement. If you aren't very, very careful, you can overlook one and wind up being reported as being in default without realizing it (this happened to me). It's a major pain to keep up with the paperwork and requires almost daily effort (difficult to do as a resident). Granted, there are ways to get back out of default, but those usually involve the lender adding fees and increasing interest...hence the motivation to try to get you into default.
I would advise anyone to do their best to minimize student loan debt. THis is why I'm an advocate of state schools. The quality of your med school education depends much more on your effort than on the name recognition factor of your med school.
You don't want to wind up making a career choice based on whether you can pay your loans. I would also advise to NOT buy new cars or large homes until you've at least made a major dent in the debt. Paying these loans is very, very painful. Anyone going though med school should be a master in delayed gratification...increasing this now (and delaying a few years as an attending) can help you greatly down the road.