I think that's true in the sense of moving down a ranking list (or not getting ranked at all). We've had cases where on paper, an applicant looks phenomenal, but then we meet them and there are red flags. I think it's harder to move up, though. I think someone can better communicate fit in person than on paper, and so that person can move up quite a bit, but I don't think I've ever had someone from the bottom third move to the top.
I wouldn't put a lot of stock in what getting a "second wave invite" means. It could mean that your application was assigned to a faculty member who was slower at reviewing and so you weren't even considered for the first wave. I also agree with WisNeuro. Even if you may be further down the list for a site to begin with, it doesn't mean you can't move up and it doesn't mean you can't match there. There are excellent sites who will even go on to have openings for Phase 2.
All this being said, an interview CAN help you and it CAN hurt you. So I would still go into it putting your best foot forward. Have a good understanding of the site and why you're interested in it - I think candidates who do this are able to communicate fit better than candidates who don't prepare.