I would be nervous about that as well. But they do a 3rd interview in the case that your original 2 interviewers scored you more than 3 points apart, so as long as you only had 2, either A) he scored you better than you may have thought, or B) your other interview didn't go as well as you believe it did
I have that same worry, though. One of my interviewers was more serious, and the other and I talked for a while rather casually about mutual hobbies. But no 3rd interview, so they must not have been too far apart on what they thought of me.