Don't listen too closely to SDN dogma. Be realistic to what you know. We have a close analogy in the application process from UG to med school, use it to feel out this decision. Of course school reputation will matter, but not because of differences in education (likely to be small) but because of what it says about you to have gotten into a top or bottom tiered school in the first place. But that is the purpose of standardized testing - to allow the kids who went to ASU on a scholarship to prove they can get a better MCAT than the kid who coasted to Stanford.
A few ideas:
1.) For the vast majority of cases, it doesn't help much to have gone to a top school. You get a slight benefit of the doubt, a raise of the eyebrows, but that's easily offset by the higher caliber of students you'll be competing with. Make no mistake, there will be differences, and it IS better to be a slightly bigger fish in a smaller pond.
2) But there are exceptions - think I-banking and consulting - it is taken as a prerequisite on wall street that you have gone to a top school where they recruit, with relatively few exceptions. The same may be true of top programs in top specialties. Probably not many, but do your research. I'm sure it exists.
3) Only the very top (top 15?) and the very bottom make an impression, almost ever. Programs that are somewhere in the middle and especially state schools are probably all about the same in most people's eyes. Going to a solid state program to save money and kicking ass will rarely be held against you. Going to Johns Hopkins or Harvard may make a difference in your career, but going to USC instead of UVM or UA PHX? Doubt it.
Obviously people here are not willing to admit that everyone has a threshold of what feels not good enough for them. Many would not go to the caribbean, how is that any different? And the argument that you shouldn't have applied anywhere you didn't want to go is moot. You did, and sometimes you don't know until you're there. Do what feels right. Investigate the extent to which schools you've already applied to (who will know that you had an acceptance come March via AAMC rules) would search that database next year and/or even care that you had previously rejected an acceptance. That hasn't been made clear to me ever.
And consider your life circumstances -- I believe there's a lot to be learned in the world by spending a few years and being a nontrad, so don't listen to the idea that you'd be "wasting a year of your life". Since when did not being a doctor mean your life was wasted?
Good luck!