Diversity encompasses many factors, and race and ethnicity are just two. People from every ethnic background in America experience poverty and barriers to social mobility in some form or another. It would be naive to argue that a black doctor is better able to empathize, connect, and care for another black patient if their socioeconomic, political, religious, class, etc. backgrounds differ to a large extent versus let's say another Asian doctor shares that background with the patient. I've worked with (not shadowed, WORKED) with physicians in a community that is incredibly diverse. I don't recall a patient feeling uncomfortable because of a physician's race, but hey, I live in California, which is probably a bubble. (<-- if your next comment is, well I don't live in multiculturally diverse states like Cali, NY, Washington, etc. touche, I probably lost that argument in your state

)
Let's not reduce AA, its flaws, and its merits to a simple equation, because it isn't. Just because AA isn't perfect doesn't mean we can't improve it. That kind of thinking gives AA opponents greater strength. These kinds of programs REQUIRE transparency. I am not against AA. And "small" advantage is subjective; I would argue AA provides a "large" advantage based on numbers and statistics in California.
If all applicants who benefit from AA (doesn't matter what ethnicity) were to guarantee service in under-served communities, I would have NO problem with it. If all stats and activities of applicants of AA were readily available to the public (anonymously of course), the public wouldn't have reason to worry.
My 2 centz...
*EDIT: I agree with your last sentence, however. Diversity does improve team-based capabilities, but again, diversity is not limited to the shade of a person's skin.