I actually just went through a similar situation. I applied for the army one and was pretty far into the process when I decided to relinquish the scholarship. The main factor was that I could not make such a life-altering decision that would potentially not even affect me for 8+ years (ie. assuming 4-5 years med school w/research year, then doing at least a 3 year civilian residency if so chosen) at the time. I thought back to 5 years ago when I started undergrad, and how I never even imagined I would be in med school next year (I probably would've thought grad school even 3 years ago). And I thought from day 1 I knew exactly what I wanted to do. Point being that I cannot imagine where I will be over the next 10 years, and the thoughts of having a family and wanting to choose where I live are too much for me to want to limit that now. I'll be honest and admit the prospect of financial freedom was enticing; however, after much more thought there is no way I can justify that as a compelling reason. I agree there are benefits to the program, including having both an army and civilian match, seeing different parts of the country/world you may not have otherwise, fulfilling your duty to your country, but I was afraid to do something I will later regret. The army will always be there, and if this is something that is still attractive while you are further in med school or beyond, your involvement would still be welcomed.
I was hesitant to tell my recruiter at first, but he was more than understanding. Good luck with this difficult decision.