Who are the people saying this to you? Do they know much about nontrads and/or have insider information based on serving as an adcom?
My experience has been that most premed advisors do not know how to advise nontrad students. The more nontraditional you are, the more this is a problem. I applied at age 30 with a 4.0 GPA for my PhD, a 43 MCAT, and no UG grades/GPA/credit hours from my hippie LAC. I was teaching gen chem and organic chem; most of my students were premeds.
You know what advice I got from the friendly university premed advisor? Retake the prereqs for grades. Seriously. She thought I should go from teaching chemistry classes to premeds one semester, to taking chemistry classes alongside my students the next.
Well, there's only one sure way to definitely fail, and that's to never apply.
Stop right here. Comparing yourself to other people is a game that you will always lose, every time. There is nothing in this world that you are the very best at. That's ok. You can still be a competitive applicant without being perfect. Adcoms aren't looking for perfection in applicants.
No, it's not near impossible. Granted, there are no guarantees with med school admissions any more than there are guarantees with anything else. But there are certainly many things you can do to greatly increase your odds of getting into medical school. The reason why most people follow certain tried and true paths is because they are more likely to work. Try to take things one step at a time instead of worrying about it all at once. Right now, you are early on in the process. Your job this semester is to ace your post bac classes (or as close to it as possible), and to get some clinical experience. That's just two things. You can do two things. Don't even think about the rest of it right now.
I'm going to pretend you never even wrote this sentence. See above.