I was in the same boat as you because I took all really hard classes at a difficult school and I was afraid they wouldn't recognize it. As I was told by CSU, however, they can tell if your program is rigorous by your GRE scores and your writing quality. So, I went out and got exceptional GRE scores, wrote really good essays, and wound up getting accepted (although, not to CSU...hmmm). Do what's best for you and don't feel like you have to play the "admissions game". I thought I was doomed for sure, but now I'm starting to feel like vet school admissions is hyped up to be harder than it actually is (not that it's easy by any means). As long as your GPA isn't in the dumps, and the rest of your application shows that you're prepared, you'll be fine.
As a side note, the VMCAS gives you 2000 characters (I think) to explain whatever you want. If you feel it is necessary, you can (briefly) mention in that section that you took the hardest classes available, and I'm sure they'll take that into consideration.