Let's assume they do talk (while they definitely do after May 1, it is speculation as to if they do before then), do you think this would hurt your chances? I think not. If the other schools say that you are not one of their top candidates, Penn knows that if they accept you, you will attend. Their matriculation percentage goes up. But Penn has a good enough rep and program to accept who they want based on what THEY saw and still have good matriculation percentages.
Basically, don't worry about the other interviews, just about Penn and if it is a good fit for you if/when you are accepted.