I'm fairly certain that my medical students fear me. It's not intentional; I just like to pimp a lot for teaching on rounds and I always start with the students. I also tend to be pretty serious on service. Also, my service is a place where people have frontal lobes herniating through their noses, so the intimidation level is already kinda high.
My point is, although I may appear frightening, I would never submarine you with a bad LOR, and if I thought you really sucked, I would politely decline the request to write one (has never happened).
Maybe this dude is just a little hard to get to know, or ascerbic, or whatever...or maybe he really is a jerk. Either way, it takes a real sociopath to sandbag you with a bad LOR just because he doesn't like the way you tie your shoes. Ask him if, given the time he has spent with you, he feels comfortable enough with you to write a supportive letter. He's not going to lie to your face. If you really, really don't want to ask him, then you're going to need to try to resurrect your neuro experience with some time spent with a different attending(s).