I am in the situation you describe concerning letters of rec. I went to this prof's OH a couple times during the course but I doubt he really remembered me, although I did well in his class. Frankly, if that's all you've got, you just gotta give it a go. I asked him for a LoR. anyway, and he agreed. He actually told me to write a draft of a LoR for him, and I'm gonna meet with him to talk further in depth next week. It helps to make a packet for the prof as well, like whipping up a personal statement and giving them your transcript and resumes, etc. All things considered, I think things have worked out well. Obviously its best if they knew you for a long period of time, but when it comes down to it, ya gotta do what ya gotta do. Best of Luck!!