This. My main reason was exactly the same.
What stage in med school are you in? Neuro was really cool in 1st/2nd year, but I thought that real everyday neurology was just tedious. Check some tone, power, sensation, reflexes, localize the lesion, try to convince the student that there's some benefit to doing all of that instead of just looking at the MRI, then refer to occupational therapy.
Also, if you haven't done a surgery rotation yet, then that'll tell you whether you want to consider neurosurgery. Either surgery is for you or it's not. I don't think I've ever met somebody who, after doing both rotations, was still debating between psych and neurosurg. You may be treating the same part of the body (which makes 1st/2nd year fairly similar), but it's like debating between chess and football because they both involve strategy.