Everyone relax. What they mean when they say you are evaluated over the course of the day is that anyone you interact with is welcome to provide the committee with information about you. If someone thought you were "that guy" and cared enough, the could approach the adcom. Likewise, if someone you ran into thought you would be a perfect fit for Duke, they could also approach the adcom. There is no nefarious plot, no crazy hidden recording devices, just a way for people who you interact with during the day to have some input.
In short, if you were generally polite and respectful to everyone (applicants, staff, physicians, etc.) on your interview day, this will not effect you at all. If, on the other hand, you punched someone in the face, then you have reason to worry.