I had the same problem a couple years back. I had this feeling that I should go into neurology, since I like the neurosciences so much. But, my real love was with psychiatric illness.
As far as choosing which field, nobody can decide but yourself. But, consider that there are many psychiatrists that focus heavily on the biological aspect of psychiatric illness, and do no psychotherapy at all. Many are researchers that don't even see patients at all. In the fellowships sticky above, you can see that as a psychiatry resident, you are eligible for either behavioral neuroscience fellowships, as well as neuropsychiatry fellowship training. This will allow you to study both psychiatry and neurology in depth.
Remember that psychiatrists are boarded by the American Board of Psychiatry and Neurology. 40% of our boards is neurology. 40% of their boards is psychiatry. The fields are heavily intertwined. Think about what kind of patients you want to see primarily, keeping in mind that there is often overlap. When doing this, think of the kinds of treatment you'd rather be giving. i.e. stroke patients with tpa, neuro-rehab and the like; along with migraines, neuropathic pain, spinal cord diseases, etc. This is in contrast to psychiatry, with schizophrenia, personality disorders, mood disorders, neuropsychiatric syndromes, psychiatric manifestations of medical illness, being treated with ECT, antipsychotics, older neuroleptics, and in the near future, VNS or TMS.
If you're truly a glutton for punishment, do a combined residency in Neuropsychiatry. It's 6 years though, I think. Details can be found also in the fellowships thread sticky.