I would make your decision primarily based on how much you want to truly be a doctor. Besides what you've listed, biggest difference I have seen between the two at Harvard it is expected you will train, and that has a big influence on the culture. Stanford is a great place to go to medical school if you don't want to be a doctor.
Besides the culture difference the main difference is the level of affiliated hospitals. Harvard is like UCSF and Stanford merged (e.g. best hospitals in the region + stupid number of researchers + lots of business and other opportunities). That can be a pro or a con as you've noted as it leads to less class unity and it's easy to get lost in the sauce. As far as not starting research M1, not sure who told you that. You have a good amount of time for non-classroom activities first year, as only ~6 hours/day M-F are needed for class + prep. Many people decide to use that for things outside of research, but many also are very productive at HMS M1 year.
Overall you can't go wrong, but I'd think a lot about the culture you want. If you're going into medicine because you want clinical practice to be a part of your career and you think good research or innovation should be informed by actually understanding the practice of medicine, HMS would be a good fit. If you're mostly interested in the research/innovation for its own sake, and could take or leave clinical practice then Stanford would be the move.