I would go, you can always transfer if you hate it. I had/have zero regrets about having gone to ross!
Also, your coworkers are the exception not the rule. The main stigma out there about Rossies are better at certain things than US grads. The most common ones I heart were that we have better surgical skills and better client communications (since we take communications labs while most state schools do not). While I was in clinics I had many clinicians that would be like "WHERE IS THE ROSSIE because they're always better at [insert random surgical procedure/knowledge here]." I had no problem getting a job and had many competitive offers and other practices soliciting me. At my clinic 4 out of 10 doctors are from Ross.
Plus, if you're flexible on location SO many places are hiring, you will be able to find a job no matter where you go to school for that reason alone.