I'm finishing up too, and had the same question. At 2 places, everyone met with the program director, but at all other programs, a few people interviewed with the program director and the rest interviewed with other faculty. I had the same question as you, and don't know the answer for sure, but I posted it somewhere on here and people said that at some programs it probably is the stronger applicants and at some places, the program director interviews the candidates who have taken time off or have some irregularity on their record so they can hear and understand their story. Like you, I didn't really interview with the program directors too much except for the one place.
I also heard from someone at one of my interviews that there are special days at some places for applicants who the program really, really wants, like people who have done significant research. She said that alledgedly, Boston Children's has such a day, and hosts the applicants for 2 days. So yea........I haven't been invited to any special interview days or interviewed w/ program directors.