QoL probably gonna be way better at Brown, they're a happy bunch of students.
Going Harvard would probably open a lot more MD options up though. You'll have to deal with the full pains of the premed process but could likely end up at a considerably stronger MD program than Alpert.
Few statistics and thoughts here:
Harvard has an overall admit rate of >85% to medical school for its undergraduates. Princeton sends 1/3 of it's successful premeds to top 10 schools, so let's assume Harvard is similar (I can source both of these if you really need me to). If you go to Harvard, it stands that you have an excellent chance of attending a medical school, and a fairly good shot (depends on your MCAT and ECs) at going to a top program. However, as efle pointed out, you would have to go through the whole premed shebang, meaning acing science classes, worrying about GPA, doing all the necessary ECs, getting letters of rec, acing the MCAT, and all of the associated intangibles. In the long run, if you're aiming (for whatever reason), for a top tier medical school, etc, then Harvard would give you the better shot, though it
is a gamble - there's always the chance that you will be in the ~15% that don't make it, or you may end up going to a "worse" medical school than Brown. It's not a particularly safe bet, considering the opportunity cost, but it's not an enormous risk either.
On the other hand, Brown would give you the freedom to do whatever you wanted during your undergraduate years along with the essentially guaranteed opportunity to go to a wonderful medical school. This means you don't have to worry about all that other stuff, your GPA won't be as important, you have fewer hoops to jump through, and you can spend your time on what you really want to do. Your quality of life at Brown will not be better than Harvard just because its Brown (honestly both have a great QoL for most people) - it will be because you have a lot more flexibility, which is something to take very strongly into account, considering how important these years will be to your personal and professional maturation. Additionally, you are not locked into the PLME program. You are always free to take the MCAT, and if you score exceptionally well, you can opt-out of PLME and apply the traditional route (assuming you have the GPA and experiences to back you up), and Brown has historically strong placements at top medical schools as well (not quite as good as Harvard, but very very good).
It's up to you to determine what is important. Both schools will provide you with some of the best educational experiences in the world. You can't really make a "wrong" choice here, in my opinion. You are likely to succeed in both cases. It all depends on what you're comfortable risking and what you think the best path is for you.