It is definitely not good. However, it really depends on the reason for the referral. Was it because you didn't meet physical fitness or weight standards? Probably not a fatal problem for a physician to major. (No intended irony as I read this a second time.) Concern about professional competence? Big problem.
If it is viewed as a "one-off" occurrence, and you have a strong record otherwise, it shouldn't prove fatal for major. On the other hand, if there is an indication that it might be the start of a continuing problem, then it could very likely result in a non-select.
If you can have another OER after this to show that again, this was a single occurrence, or a personality conflict, then you can survive it for major. Especially since, unless things have changed dramatically, that is still a "fully qualified" board.
To give you a bit of hope, I know of a former AF medical corps general who was trashed on one OER as a captain in the 70's. He obviously survived. The disclaimer of course was that was a very different era.