Nah, that's not likely. Not to sound elitist or anything, but non-Yalies have no clue what we're like and our capabilities. I'm not even too sure what I'm up against most of the time, so taking science classes is always an "oh crap" bet. I'm sure H/P are the same. Our premed attrition happens virtually all during freshman year when they realize it's not what they want to do, not because they can't make the grade.
As mentioned above, even if it does happen on a large scale, which I'm sure it doesn't, that means other schools face even higher problems, and so all of our scores decrease and I still don't see how that changes anything.
Also, this average of 34 isn't advertised widely to the public; it's only for internal use to guide our own applicants, so there's no PR issue that they'd try to inflate the scores. It'll only harm us applicants, and that's not something they want to do because we treat our precious 95% acceptance rate highly.
It's funny. I don't believe this subject will ever be in controversial, and we all only tend to side with the side we're most familiar with, therefore leaving the others to accuse that we're biased they have no clue and vice versa. This will forever remain an Ivy vs. non-Ivy argument, the the former will be looked upon as arrogant and condescending when all we're doing is stating the truth because we go to school with some scary people.