To expand on my original post, it's highly unlikely that someone will get many interviews and not match. However, you are an N of 1, and even if .01% of people end up in that boat every year, you can still be in that .01%. As others have stated, in my hypothetical example in which you're ranked #5 by every site, it's improbable that every single person ranked 1-4 also listed that site highly enough to match ahead of you. I do believe that a few people each year simply are unlucky and end up not matching because of the way that the numbers play out; you can modify my example and just say "you can get 10 interviews, be ranked highly by each site, and still just miss the mark for matching at any one of those sites" and you see what I mean. My highly simplified hypothetical was just meant to illustrate my suggestion not to take any one of your sites for granted by turning down an interview.
If you have a lot of interviews, once the interview process if over, I don't think it's worth worrying about the incredibly tiny odds that you won't match. It's out of your control, anyway. And I'm one of those people who was in the opposite situation, where I received far fewer invites than expected, but matched to a highly competitive site, so I actually think that worrying about something that's out of your control isn't helpful regardless of your situation. But when you're still at the point where you do have control - when you're deciding whether to decline an interview - I would encourage people to attend whenever possible.