It's because of polarizability. Noble gases have a complete octet and do not interact with other atoms covalently or ionically because they possess a stable electron configuration. Thus, the only interaction they will have between each other are van der waals attractions, which are enhanced by the size of the electron cloud. The larger the electron cloud, the more readily instantaneous dipoles are created, and thus the greater the interaction between molecules. The greater the interaction between molecules, the larger the boiling point is. So yes, the larger atomic radius the larger the boiling point among noble gases. It has nothing to do with electron affinity...
