It is definitely not good.  However, it really depends on the reason for the referral.  Was it because you didn't meet physical fitness or weight standards?  Probably not a fatal problem for a physician to major.  (No intended irony as I read this a second time.)  Concern about professional competence?  Big problem.
If it is viewed as a "one-off" occurrence, and you have a strong record otherwise, it shouldn't prove fatal for major.  On the other hand, if there is an indication that it might be the start of a continuing problem, then it could very likely result in a non-select.
If you can have another OER after this to show that again, this was a single occurrence, or a personality conflict, then you can survive it for major.  Especially since, unless things have changed dramatically, that is still a "fully qualified" board.
To give you a bit of hope, I know of a former AF medical corps general who was trashed on one OER as a captain in the 70's.  He obviously survived.  The disclaimer of course was that was a very different era.