Your criticisms of EP are very valid. Basically sociobiology came along a few decades ago and brought to the forefront a synthesis of evolution and animal behavior. Not just the attempt to study why animals behave the way they do in current context, but why they evolved to do so. The development of sociobiology led to two splinter groups; the behavioral ecologists and the evolutionary psychologists. Behavioral ecology is an immanently testable and scientific discipline, whereas EP is exactly what you said, a largely speculative field. Early on, behavioral ecologists mostly concerned themselves with non-human animals, while evolutionary psychologists focused on humans. Behavioral ecologists trained in evolutionary biology and ecology, whereas evolutionary psychologists didn't train; they just decided they wanted to talk about the evolutionary roots of human behavior. BEs were firmly rooted in darwinian principles and scientific methodology. EPs were not. In recent years, the field of Human Behavioral Ecology has developed, as an offshoot of behavioral ecology. It too is a testable and scientific field, although limited in the same way that all behavioral ecologists are limited (we can't go back in time and see how H. ergaster behaved).
So to answer your question, no I wouldn't necessarily say that there is a rigorous section within EP, but there is a rigorous section of behavioral ecologists devoted to the study of the evolution of human behavior.
I do agree that factors outside the individual need to be considered, but this doesn't strictly have anything to do with evolution and is perfectly compatible with current scientific paradigms. It's just hard to study properly, so we don't do it much.
We can quibble over it but I'd have to disagree. Evolutionary biology and ecology are the de facto fields in which the individual is the unit of study, not internal processes within the individual. The biomedical paradigm does not have much room for understanding individual-environment interaction and it is shown in the language we use and the way we approach models of disease, endocrine systems, and genetics.
These factors ARE hard to study and it's harder to get grant money to do so. But that isn't an excuse to dismiss them out of hand and not incorporate them into the models we use for understanding health and disease. Unfortunately, that's exactly what the biomedical model does. Every time I hear about the 'genetic predisposition' my race has for HTN, CHD, and DM2 I start screaming at the TV. It's interesting that this 'genetic predisposition' is twice as big for those of us in this country than it is for those of us in India, and that our 'genetic predisposition' even in India is 5-10x as much as it was even two generations ago. Either the selective advantage of being 'predisposed' to all those horrible diseases leads to a 2-5x selective advantage, or its not really a 'genetic predisposition' at all.
We are still studying the genetics and pathophys of metabolic syndrome and using absurd amounts of money to do so, even though the most basic analysis with the simplest evolutionary genetic formulas show that this can't possibly be an issue of internal genetic and hormonal processes gone awry, but a mismatch between the human body and current environment.
The biomedical model holds cholesterol as the reason behind CAD and stroke, but even moderate exercise has far more effect on primary and secondary prevention than the most potent statin, despite only modest effects on LDL. So what exactly is exercise doing? WE DON'T ACTUALLY KNOW!!!
The biomedical mindset seems perfectly happy with this. Sure it's harder to study. Sure it's more difficult to integrate and understand. Does that excuse our seeming ability to ignore it altogether?
Behavioral ecology and evolution are built around understanding these relationships. The biomedical model is not. If the biomedical world used an ecolate frame of mind, this wouldn't be the case.