You are (mostly) wrong.
Here's the scoop. If two applicants want to trade spots, and two programs are truly happy with the trade, then really there is no problem.
However, that's pretty unlikely. What's much more likely is that Matched Resident A and Matched Resident B decide that they would like to switch. They now contact me (let's say I'm PD at the program where Resident A has matched) and ask if it's OK. That puts me in a tough spot. Let's say I don't think Resident B is acceptable. I would need to say no, and know that Resident A will now be unhappy. Or, resident A could say "approve the swap or I'll resign anyway".
You may say, all of that would be fine. But what about the other way? Let's say I match you, but then in the scramble is someone great from my medical school. Or, someone who was higher on my rank list calls me and says they feel like they made a mistake and wants to swap. Now I call you and tell you that I need you to swap with Resident B. And, you worry that if you don't, I'm going to make your life miserable. Maybe I will, maybe I won't, but you'll worry about it and might feel pressured to swap even though you don't want to.
So, in general the NRMP only waives matches if there is an overriding reason to do so. If you've had a major life event, etc. If you completely change your mind about what field you want, they might give you a waiver BUT it's a match violation to look for a new spot first -- so you'd need to resign your spot, get a waiver, and then look for a new spot (that might or might not exist).