I understand you completely about not wanting to be the 1 or 2 urm in a class. I was the only one in my high school and i'll never go back to a situation like that again.
One thing that helped me decide was going through the MSAR book and making a list of schools that had AT LEAST 10% URM in their student body excluding the HBCUs. then i narrowed it further based on location because i was funding my application process with the money saved from my summer job. I knew I wouldnt be able to afford any plane tickets to interview far away. then in the end, it came down to "would i actually go there?" school name didnt matter much to me compared to student body vibe and surrounding city.
The school I chose happened to have 14% URM.