There's more factors to play than just the OP's strengths and weaknesses. A major as difficult as astrophysics may have unprecedented problems, and I'll share a major ones:
Your professors in "these" courses likely don't care about your professional goals, meaning they are less likely to give lenient grades.
I had a chemistry 600 level class where NO ONE earned an "A". The professor didn't see it as "impeding" on students chances at medical school, it was just his way to allocate grading he felt was fine. This wouldn't happen in a typical pre-med course like microbiology.
There is a reason engineers generally have closer to a 3.0 than to a 4.0, and its because these professors are designed to grade this way, whether or not you are performing academically better than a typical pre-med counterpart major.
If the OP has great math/physics skills, then fantastic. But unfortunately, you have to acknowledge the University he is attended and how they grade such courses, and also the professors who teach it. You can't really prepare for this, and therefor the most logical approach is the safe and easy biology route.
I do understand what you're getting at, I really do. I took courses typical medical students would probably get C's, D's, and even fail. But even getting a B+ in those courses hurts more than it helps.