Usually, the references will give you a copy. Thus, waivering to view your letters is very common.
I think the admission committee may find it odd if they notice it. Since, they have so many information to look at, they may not even see that you did not waiver it.
I guess the bottom line is, since it's already done and you can't change it. Don't worry about it. There's so many aspects of the application that is taken into account and this one thing isn't a huge factor in the review process.
Keep yourself busy w/ hobbies, so the wait won't drive you crazy.