Based on the fact that there's no detectable increase in graduation rate since the advent of holistic admissions. Maybe even a slight decrease. Maybe graduation rate is not the right metric, but I think any other metric would have the same problem. Even if "fit" were real, and it helped, you could never tell. You could never even design a study that would accurately measure success rate based on fit, it's just too nebulous and there are too many other factors.
You might still believe that it's meaningful even if the evidence doesn't show it -- I'm aware that absence of evidence is not evidence of absence. My point is that it's just not possible to find enough evidence to make an affirmative statement that fit is real. If you still think so, then that's fine but it's just an opinion. I personally think that when ADCOMs hold that opinion, it opens up the system to being gamed by people who are good at acting, and undermines the integrity of the process.