This. It isn't that it is outrageously difficult. Nothing that I've had to learn has been hard because of the complexity of the material. Sciences are GREAT that way, in that once you understand the principles, the answers are suddenly very unambiguous and intuitive. In some sense, this makes it much easier than humanities, where there are a lot more opinions than facts.
The hard part is getting to understand and apply the principles. That takes time, and a lot of very dedicated, careful thinking. That requires discipline. If the desire to "be a doctor" isn't so great that it can overcome the desire to do every other thing that you could be doing instead, keeping focus will become impossible. That is what takes out the vast majority of pre-meds that switch paths, not inability. I've known some fine doctors who weren't necessarily the brightest people I'd ever met... but they made up for it by having great study skills (aka discipline to keep at the material until they got it.)
EDIT: And personally, I kind of envy those folks. My discipline level is evidenced by my frequent SDN postings. It takes a lot of effort for me to stop procrastinating and get my nose in my books. I'm just very fortunate that once it is there, the info sticks pretty well. Otherwise, I'd be up a creek.