What I particularly dislike about committees it not only do they sometimes bar people from applying (or at the very least, sabotage) by sometimes limiting who can receive a letter, but then try to make determinations about whether student X is a good applicant for medical school.
Medical schools are so varied in their goals/mission some are research focused, others are service focused, etc.) and it is ultimately the purpose of the adcom at each individual medical school to make this decision about an applicant - not a bunch of random people making up the committee.
Even disregarding that, you still have the problem of a small sample size of people making judgments about applications with committee letters. If person X at HMS doesn't like your application particularly much, well then yeah you are probably rejected from HMS. But that info (about HMS person not liking your application) is not broadcast to, say, person Y at Yale or person Z at UCSF. Thus, persons Y and Z can make a judgement for themselves about the merits of your application.
But if you have a committee (of say n=5) making a judgement and they dislike your application, well then that negative information is broadcast to ALL your schools. Now everyone gets to hear about how your committee thought you were average. But maybe the committee's opinion is unrepresentative (for example, maybe you had a bad day for your committee interview, maybe the particular committee people just didn't click with you, maybe the committee has no idea what to look for, maybe the committee is biased by the accomplishments of the peers at your schools). Too bad. You are still screwed for all your schools.
If you didn't have a committee letter and just bombed the HMS interview, you just get rejected by HMS. But it won't affect everything else.