You make some good points, but I would venture to say, any spots taken by anyone, whether qualified or unqualified, is a spot taken from someone else.
Med school seats stays relatively stable, so yes, someone is being left out when someone else is let in. I'm sure there are URMs that are qualified for positions in med school but your quote that most URMs you know have above average stats would put them in the minority of URM matriculants as told by AAMC:
http://www.aamc.org/data/facts/2005/mcatgparaceeth.htm
I pointed out the comparative stats for whites/asians/blacks/mexicans to highlight some inequalities in the stats of matriculants. It's in my above post upthread if you are interested.
I also understand the need to increase URM doctors is to increase the number of physicians in underserved areas, but I think the problem with AA and allowing URMs in with lower stats is that it makes people resentful when there's already limited spacing and it does URMs a disfavor because some may have actually been qualified to get in, while others may have been unqualified to enter, thus making it more difficult to catch up once in school.
The key to increasing minority involvement in the sciences isn't to lowering the bar, it's the channel funding into programs that will encourage URMs, poor minorities to stay in school and to generate an interest in the sciences in them, and hence become competitive for med school etc.