Absolutely not. There are a finite number of interview spots. (most places I am familiar with offer about 150 interviews). We had almost 700 applicants. Waitlist is at least 100 deep. Many of them in the same 'competitiveness' pool. It just isn't feasible to interview everyone. So, if you get an interview and its a place you like, go. Have a stellar interview. I have seen people on the waitlist interview and then match. Don't put your ego to much into this. If you really wanted to interview at the program, really like the location etc and you get an interview, take it.
If it was just one of many, isn't high on your list then you might not want to 'waste the time.'
If you do take the interview, I would highly recommend not going in with that chip on your shoulder. A seemingly disinterested interviewee will be dropped lower than an friendly, interesting open one.