I agree with aprogramdirector.
I doubt they will even care. The only way they probably would remotely care would be if you went to great lengths to beg to do this elective with some very important person, and then backed out...they might wonder why. If you personally know the person you were going to rotate with, you might want to go mention it to them that you had to rearrange your schedule...or say something vague like that. I didn't realize this when I am in medical school, but attendings really don't even notice medical students most of the time...you are like ants to them. The only time they notice you is if you really pissed someone off, or if you maybe did something really brilliant.
It's important to be happy in what you end up doing, so just change your schedule if you need to. I'm 98% sure the medicine department won't care. If they do, they're probably kind of petty and you might not want to do medicine there anyway!