I think it's doable, it depends on how much you want it and are willing to work for it. Time-wise, undergrad in 3 years is completely doable; keep in mind a lot of it is stuff like English, Poly Sci, Women's Studies etc., at least at the beginning. I did it in 3 years and got through it.
If money is a concern, you can always go to a state university and supplement with taking some of the general ed courses at a community college, which would be cheaper. You don't necessarily have to start at a CC and then switch either; you can be taking online english or general ed classes at the CC while concurrently taking your Bio and Chem classes at the university.
First two years of med school were OK; the second year was much better than the first. I worked on the side (research, tutoring) which made it more interesting and fulfilling. Then you have to get through Step 1, which is tough, but if you plan well, it's again very plausible to get what you need to do neurology (ave score in the high 220's?). Definitely not easy by any means, but a little work every day for several months goes a long way.
Clinicals have been simultaneously much more interesting and boring (sounds paradoxical, but I mean at different times) than the first 2 years. Sometimes things move fast and are exciting, a lot of the time you're just waiting around--it's a roller coaster. Studying seems like it would be harder, because you have less time, but so far it's been easier. If you've done the first two years well, then you've laid the foundation and when you read, you keep having a voice go off in your head that says "Oh yeah, I remember that" or "That makes sense". You go a few steps further than you did the first two years, because you learn about diagnostic tests and treatment plans for diseases. The hardest part for me is learning clinical skills that only really are cemented with practice, which I feel I'm getting better at as time goes on. I don't want to understate it; it hasn't been easy for me, but I think I'm "staying afloat" and learning what I'm supposed to be.
I can't speak to residency, as I'm not there yet. But one thing you could try to do, aside from shadowing, is see if you can spend a day at a medical school in your area (assuming there is one) seeing what the first two years are like. We've had some undergrads sit in on our lectures and go through the day with us.