In my opinion, no. Your future bosses are likely more concerned with where you did your neuro training. Just be sure you know why the programs around you aren't good. If they're less desirable because the residents are all miserable, that's worth considering.
On the other hand, if you do your prelim year close to home, and do a neuro elective, it might be helpful in terms of landing a job back there a few years later.
I recommend doing a prelim year at a program where you'll have solid medicine training, especially if you'll be doing your neurology residency in a hospital with sicker patients. At the hospital where I did my neurology residency, my neurology patients were medically complicated. A medicine consult service was available, but the medicine consult service was not available to co-manage internal medicine issues in all of our patients.