Well, here are my 2 cents (probably not my own but metabolized from something or other I've read).
Shame, as I understand it, is more of a long-standing, private phenomenon. It is what is internalized after a humiliating experience, and accommodated into a self-concept.
Humiliation, I'll guess, is more of an acute response to a public act in which one or more people use personal characteristics of the victim to assert power over him/her. I see it as more of an immediate or short-term response to a time-conscribed event, than shame, which I see as an internalized feeling about one's self, deeply integrated into a self-concept. I guess that in an act of humiliation, the presence of witnesses may be important. (Maybe not necessary to initiate an act of humiliation, but probably increase its likelihood, and probably heighten the drama inherent in the action, or motivate the person doing the humiliating to make more of a show.)
Or another way to put it: one has been humiliated, and so feels shame.
What is common between them: both often involve an attack on core, vs peripheral aspects of the victim/experiencer's being - his/her body, face, skin, gender, ethnicity. Or focus on details that undermine an aspirational or actual identity. Egs of targets: a boy who displays stereotypically 'feminine' characteristics (e.g., being small or uncoordinated), or a girl who shows 'masculine' ones (having more hair on her arms than the dominant ethnic group has).
Lots of times there's a sexual dimension. Remember that the guard subjects in Zimbardo's prison experiment got into sexually sadistic postures in their efforts to humiliate the prisoners, as did the soldiers at Abu Ghraib, to their real ones. (Authors in the psychoanalytic tradition may have a lot to say about that - look at things about S&M, dominance and submission.)
Edit: also, probably, ideologies frame the act, and suggest its content (as with gender discourses -> sexual humiliation).
Edit: AND, ideologies also frame the victim's memories/interpretation of the experience. If a culture has already created a space/language for a certain kind of victimhood, someone who sees himself as that kind of victim will use those schemas to think about what he felt.
I think in psych you will have more luck with 'abuse' + 'victims' than 'humiliation', in a search. Maybe a search in/around body dysmorphic disorder, social anxiety, or sexual abuse, in particular, will help you find other definitions. Also, 'bullying'.