The internet makes it seem like 90% of applicants have the profile that you describe, but the label of URM exists because these applicants are, well, underrepresented. They don't come close to making up a majority of med school applicants or acceptances.
Some data to show why being URM isn't the advantage everyone makes it out to be:
2024 FACTS: Applicants and Matriculants Data
I used A12 to find these numbers. If my calculations are incorrect, feel free to correct me.
Overall Asian acceptance rate: 39.9%
Overall White acceptance rate: 39.1%
Overall Black acceptance rate: 35.7%
Overall Hispanic acceptance rate: 40.7%
Overall Hawaiian / Pacific Islander acceptance rate: 34.1%
Overall American Indian / Alaskan Native acceptance rate: 43.8%
Overall acceptance rate: 38.0%
Holistic admissions may not view you the same as a Black, disadvantaged applicant, but medical schools have more than one seat to offer. For every one Black matriculant, there are eight White / Asian matriculants.
This comes back down to what you initially said - a lot of this comes down to things you can't control. Focus on what you can control and hope it works out. I know SEVERAL 'average' applicants that got in this cycle to good schools. If you can find a way to make your experiences and personality shine I'm sure you will be joining them this time next year.