I believe that this is a rumor that has completely taken on a life of its own. I interviewed at Duke Neuro even though I had been told by others that it was a malignant program. The interesting thing is that the 5 people who had told me about Duke's malignancy had never interviewed there themselves. They were just passing on info that they themselves had heard from others. One guy even told me that they don't let you meet the residents.
Well, I met the about 10 residents and some of the faculty. The residents do work hard, but I don't think they work any harder than the residents at Wash U, Emory, Baylor, Cleveland Clinic, etc. The program director was a pretty nice guy. Also the neuro boards passing rate at all the above programs is 100%. I was told that the national passing rate is around ~70%.
The best compliment I can think of is that I met two people who had done away rotations at Duke and both were ranking Duke #1. Another student whose opinion I regard highly also ranked Duke in her top 3.
About it being a department within medicine: I didn't really sense any difference with respect to the life of the residents. The big difference is at the top with resource allocation and with the faculty. The residents see patients, do their research, and graduate, just like at all the other programs.
I personally liked the program, but could never see myself living in Durham.
My advice to you is to go down and see programs for yourself. There were many rumors on the interview trail that I later found out to be untrue. Also remember that we all value different things in a program.
Good Luck.