1. You don't know what you want.
You are a college Freshman. You don't know what medicine is. You don't know what a Psychiatrist does. You don't know what other options are out there within healthcare and not. And that is okay! That is perfectly normal. The first step in anything is figuring out who you are and what different careers entail. It is great to have interest in medicine early on, but you need to justify it. Not to admissions committees, but to yourself. Shadow, get clinical experience, not for an application, but so that you can reliably say, I know what I'm getting into and it is worth it.
2. Medical training is long and expensive.
You are dedicating a decade toward becoming a physician. And yes, you will be much busier than your average person. Schools are getting more and more expensive and loan options are likely to get worse in the next couple of years. From a strictly financial perspective, medicine makes absolutely no sense for the majority of the types of people that end up getting into medical school. You are recognizing that already, which is very important.
3. Social life
Your social life is what you make of it. Undergrad, medical school and residency are not all consuming. Even working 80 hours a week (the general upper bound), there are 40+ hours of waking time to do other things. You have less time. You can't veg on the couch and have hobbies and a significant other etc. The people that say otherwise are doing it wrong or don't belong in medical training.
4. Missing out on your 20's.
You are going to miss out on things. It is inevitable. But, this is relatively minimal until you start residency, which would be no different than if you started working in most professions. There is no "fun you're supposed to have" metric. There just isn't. I think
@Goro 's comment was about medicine, I would expand it to life in general. You are likely going to be around for 7-9 decades. Your long term happiness is not going to be tied to several years in your 20s. There is more to life than living life up while young.
5. Family.
Being busy comes at a price. But, in no way is medicine exclusive to having a family or even delaying starting a family. Does having a family while going through training make things more difficult, absolutely. Is it possible? Yes. Is being the spouse of a physician hard? Yes. Does it take a certain amount of patience and understanding? Yes. If your priority is family, then it should remain your priority during your training. You need to understand though that this may mean that you need to sacrifice other things in order for that to be possible.
6. Medicine as a career.
Medicine offers things that few professions can match. But, it is NOT the pinnacle of professional achievement. It is NOT for everyone to aspire to. The vast majority of individuals would be miserable going through medical training and being a physician. This is a universal truth, but enjoying what you do in your job takes a lot of pressure off of other areas. Why am I in Vascular Surgery? Because I'm in the middle of Surgical residency and am the happiest person that I know. I can stay here in the hospital for long hours because I love what I do. I can sleep on a chair in the ICU next to that patient that keeps coding because, despite being exhausted the next day, I feel happy about what I did. I got lucky to find something that I am passionate about and enjoy spending my days doing. But, not everyone is like this and certainly not every branch of medicine is like what I do. Do not go into medicine if you can only see yourself happy doing one particular specialty. For starters, some specialties (not Pscyh) are difficult to get into and may not be a practical reality for you. But, more importantly, people's opinions change. Their preferences change. You need to have bailout options both outside and inside (if you go to medical school) medicine.