I believe you can shadow a relative, but when it comes to asking for a letter of rec from a dentist -- I think schools would want one from a non-relative.
There is no problem shadowing your brother, I even think it's a great idea. However, I would not get an LOR from him but rather another dentist b/c a letter coming from him may seem biased in the eyes of the admissions committee.
I believe I read that OSU needs the hours to be from non-family members. I think the experience might be a little different if you shadowed a family member so schools want you to shadow others as well. Also, a lot of schools need a dentist as a LOR, so you need to get to know one any way. Nobody is going to stop you from shadowing a family member though.