Let me start with... If you are feeling down or depressed, out of place, neurotic as you describe it, confused, etc, I recommend you speak to someone at school about it - be it your advisor, a counselor of sorts, a psychiatrist, etc as they will be best equipped with knowing you and your background, your grades/standing, study habits, your schools statistics, etc to be able to help you much better than an online forum in making such a decision.
With that said, you are blowing things way out of proportion. A search that brought up the 2018 data shows that there are 557 family medicine residencies, 517 IM programs, and 211 pediatrics programs (all excluding combined programs i.e. family/IM, med/peds, etc). You don't want to be in 1 area of a particular state. There are 49 other states (plus PR if you speak Spanish) to which you can apply - that is 1285 total locations (not to mention how many positions each location has!) that you can apply to, so 1282 locations taking into account the 3 you don't want to apply to. These are also the lesser competitive fields (though depending on specific locations or types of programs) - just generally speaking. I'd be quite surprised if you're school matches people ONLY to those 3 places you are referring to as well, so inquire/take a look into where else students from your school match consistently and recently. People have failed boards (sometimes multiple times) and still match to various specialties around the US, including the fields you list as having interest in.
Good luck with making the decision.